Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands

Abstract

Learning creative tasks that involve using hands and the body, like knitting and music playing, can be challenging for beginners. We introduce Origami Sensei, a mixed-reality system that assists beginners in creating origami by providing step-by-step instructions in real-time. Origami Sensei enables users to create their own origami pieces from a library of pre-existing designs. It uses computer vision algorithms to identify the user's current origami step. It then provides them with real-time visual projections and verbal feedback to assist them in completing the next steps. The current setup consists of a tablet, a reflective mirror and a small projector. This research approach opens possibilities for introducing mixed-reality systems for learning hands-on skills in other fields such as welding, sculpting and even surgical operations. Future steps for this project include developing multimodal instructions and conducting user studies.