Mobile IoT Interface using Emotion Information and Arduino on the Android Application

Abstract

This paper proposes mobile IoT interface using data on users' emotional state and bluetooth of IoT. For the purpose, we develop an AR(augmented reality) game on Tablet PC. Joystick is used as the interface linking reality and AR in bluetooth of IoT environment. User uses joystick to freely move the virtual object in the AR game to interact with the opponent object. In particular, the camera on tablet PC is set at front mode to detect the face of the user. After that, face detection and landmark information are used to get information on user's happiness, which is then analyzed to generate data on emotional state. Here, the info on the emotional state is utilized to modify the difficulty levels of the game. The performance and effectiveness is verified through experiments. Through experiments with the designed game, we could compare happiness data with threshold. If the happiness data is larger or smaller than threshold, we changed factors which could affect interaction, to influence the user in a meaningful way. That is, in case when the happiness data was smaller than the threshold, we increased the movement information of the object in the AR, to make it move quicker so that it could defend against or even attack the foe object which was attacking the eye of the user. If the happiness data was bigger, we reduced the movement information to realize normal interaction with the enemy. The proposed method is not just limited to providing augmented information. It proposes the new interface using information on user's emotional state, and possibly increases fun and immersion for user experience. The proposed method might be used as various contents reference points for following researches in detecting emotional state of users. © 2017 SERSC Australia.