An auto-correction teleoperation method for a mobile manipulator using gaze tracking and hand motion detection

Abstract

Situational awareness in remote environments is crucial for human operators to teleoperate mobile manipulators confidently and reliably. Visual feedback is the most common way for environment perception, providing rich information to human operators. This paper proposes an intuitive teleoperation method by combining gaze tracking and hand motion detection to teleoperate a mobile manipulator. A camera is fixed on the end-effector of the mobile robot's arm to provide visual feedback, acting as the eye of the teleoperator. Rather than direct remote control of the robot, an adaptive auto-correction mechanism is introduced for helping human operators to achieve better hand-eye coordination of the teleoperation experiences. The mobile manipulator can adjust its behaviours, such as speed, while gaze and hand movements of the operator are in different states. The experiments carried out demonstrated the effectiveness of the proposed algorithm and the survey evaluation verified the practical application value of the system.