A Mixed Reality-Based Training and Guidance System for Quality Control

Abstract

The increasing demand for customized products has raised the significant challenges of increasing performance and reducing costs in the industry. Facing that demand requires operators to enhance their capabilities to cope with complexity, demanding skills, and higher cognitive levels, performance, and errors. To overcome this scenario, a virtual instructor framework is proposed to instruct operators and support procedural quality, enabled by the use of You Only Look Once (YOLO) models and by equipping the operators with Magic Leap 2 as a Head-Mounted Display (HMD). The framework relies on key modules, such as Instructor, Management, Core, Object Detection, 3D Modeling, and Storage. A use case in the automotive industry helped validate the Proof-of-concept (PoC) of the proposed framework. This framework can contribute to guiding the development of new tools supporting assembly operations in the industry.