Storytelling-based hand gesture interaction in a virtual reality environment

Abstract

This paper focuses on emotional effects of storytelling-based hand gesture interaction in a VR (Virtual Reality) environment. Unlike using depth cameras like Kinect sensor, a wearable band is proposed to detect users’ hand gestures to create a virtual reality film. The VR film is created using a storytelling-based hand gesture recognition system and focuses on the users’ emotional effects. For the design, the hand gestures suitable for the story in the film are derived from research on users and are applied to the VR film titled ‘Not Alone’. In order to recognize the hand gestures of the user, the data collected from the wearable band is analyzed to classify movements of the hand through machine learning, which helps to solve problems of the existing hand gestures. This study proposes hand gesture interaction that is most suitable for users with two free hands in the head mounted display HMD (Head Mounted Display) based VR environment and tries to maximize users’ emotional responses in narrative-based film content by developing storytelling-based hand gesture interaction.