Combining 3-D Human Pose Estimation and IMU Sensors for Human Identification and Tracking in Multiperson Environments

Abstract

Human pose estimation (HPE) based on deep neural networks aims to predict the poses of human body in videos without needing markers. One of the main limitations in its applicability is consistently identifying and tracking the keypoints of an individual in multiperson scenarios. Despite various solutions based on image analysis being attempted, challenges, such as model accuracy, occlusions, or individuals, exiting the camera's field of view often result in the loss of the association between humans and their keypoints across video frames. In this letter, we propose a human identification and tracking methodology in multiperson environments based on data fusion between HPE software and wearable inertial measurement unit (IMU) sensors. We demonstrate how to align the data generated by these two sensor categories (camera-based HPE and IMUs) and assess the alignment between each skeleton of keypoints and IMU pair using a scoring system. In addition, we illustrate how to combine different metrics, such as orientation, acceleration, and velocity, to address alignment problems caused by inaccuracies in sensor data.