Augmented Reality for Interactive Robot Control

Abstract

Robots are widely used to support mission-critical, high-risk and complex operations. Human supervision and remote robot control are often required to operate robots in unpredictable and changing scenarios. Often, robots are controlled remotely by technicians via joystick interfaces which require training and experience to operate. To improve robot usage and practicality, we propose using augmented reality (AR) to create a more intuitive, less training-intensive means of controlling robots than traditional joystick control. AR is a creative platform for developing robot control systems, because AR combines the real world (the environment around the user, the physical robot, etc.) with the digital world (holograms, digital displays, etc.); it can even interpret physical gestures, such as pinching two fingers. In this research, a Microsoft Hololens headset is used to create an AR environment to control a Yaskawa Motoman SIA5D robot. The control process begins with the user placing an interactable holographic robot in 3D space. The user can then select between two control methods: manual control and automatic control. In manual control, the user can move the end effector of the holographic robot and the physical robot will respond immediately. In automatic control, the user can move the end effector of the holographic robot to a desired location, view a holographic preview of the motion, and select execute if the motion plan is satisfactory. In this preview mode, the user is able to preview both the motion of the robot and the torques experienced by the joints of the manipulator. This gives the user additional feedback on the planned motion. In this project we succeeded in creating an AR control system that makes controlling a robotic manipulator intuitive and effective.