Challenges and Findings in Creating Smart Assistants for Mixed Reality Training Apps

Abstract

Mixed Reality (MR) headsets such as Microsoft's HoloLens have not only revolutionized the immersive experience for entertainment but also been adopted by various industries to improve work procedures. One of the particularly popular types of MR apps among industries is the training app. In this pilot study, we created and tested a prototype to expand such MR training apps from simply being a hologram manual to an assistant that recognizes if the user provides the task correctly. We first obtained feedback from participants in a user study regarding the design, usability, and experience of the app. We then fed the videos recorded in the user study to train an object recognition model via transfer learning. In addition to participants' positive feedback on the design and promising results on object recognition, we also identified unexpected but important issues for MR app design and programming, including trade-offs between the user's workflow and the accuracy of the smart assistant.