Control yourself: A mixed-reality natural user interface

Abstract

Control Yourself is a natural user interaction system with a camera and depth sensor, a processor and a display. The user’s image is separated from the background of a camera’s output and rendered in the program in real-time. The result is that a display shows a video of a person inside the application. The software also recognizes various types of movement such as gestures, changing positions, moving in frame and multiplayer interaction. The technology utilizes the obtained gestures and movements for GUI transformations and creation and for positioning the image or mesh of a user with the background removed. Users of the system can manipulate virtual objects and various features of the program by using gestures and movements while seeing themselves as if they were viewing a mirror with an augmented reality around them. This approach allows users to interact with software by natural movements via intuitive gestures.