Latency and sampling compensation in mixed-reality-in-the-loop simulations of production systems

Abstract

X-in-the-Loop Simulation methods (Model-in-the-Loop, Software-in-the-Loop and Hardware-in-the-Loop Simulation) enable the virtual commissioning of production systems in the mechatronic development process by coupling control systems and digital twins. Mixed-Reality-in-the-Loop Simulations (MRiLS) extend this principle with Mixed Reality visualisation technologies to enhance the visual fusion of reality (e.g., real environment and human) and virtuality (digital twins), opening up a simulation loop in the reality-virtuality continuum with novel application potentials e.g., from development, training to maintenance. A major challenge in MRiLS is the positioning error of actuated real-data driven virtual components caused by latency and sampling processes between the industrial control system and the Mixed Reality device which significantly limits the application scope. To reduce this error, the paper proposes a compensation method that synchronises the Mixed Reality device to the stable time base of the control system and integrates a predictive positioning of virtual components. A software-based synchronisation method is presented, which allows the online estimation of the End-to-End latency between the control cycle and the visualisation. For prediction, interpolating and approximating section-wise defined polynomials are analysed. The error reduction by applying the compensation method is shown in a realisation example of a virtual gripper linked to a real robot kinematics.