Enhancing Real-Time Body Pose Estimation in Occluded Environments Through Multimodal Musculoskeletal Modeling

Abstract

In recent years, there has been a growing interest in Human-Robot Collaboration (HRC). One of the main challenges in developing effective tools for HRC is accurately estimating human pose in real-time, ensuring both human safety and efficient collaboration. To address this, we propose a novel approach enabling accurate and robust full-body pose estimation in real-time, even in the presence of occlusions. Our system combines information from RGB-D cameras and inertial measurement units, leveraging it to control a musculoskeletal model of the human through a multimodal inverse kinematics optimization. This approach ensures improvements in the anatomical realism and accuracy of the tracked movement while allowing flexibility in accommodating various sensor configurations. The consideration of the underlying anatomical structure also enhances the ability to estimate body poses in occluded environments. We conducted several HRC experiments where the operator’s view was obstructed by various types of occlusions. The outcomes demonstrate how our methodology significantly improves pose estimation accuracy, even with a limited set of sensors and in the presence of occlusions in the scene. Our work aims to facilitate advanced HRC applications that require a precise understanding of human movement.