Multimodal Movement Activity Recognition Using a Robot’s Proprioceptive Sensors

Abstract

By recognizing patterns in streams of sensor readings, a robot can gain insight into the activities that are performed by its physical body. Research in Human Activity Recognition (HAR) has been thriving in recent years mainly because of the widespread use of wearable sensors such as smartphones and activity trackers. By introducing HAR approaches to the robotics domain, this work aims at creating agents that are capable of detecting their own body’s activities. An activity recognition pipeline is proposed that allows a robot to classify its actions by analyzing heterogeneous, asynchronous data streams provided by its inbuilt sensors. The approach is evaluated in two experiments featuring the service robot Pepper. In the first experiment, a set of base movements is recognized by analyzing data from various proprioceptive sensors. The findings indicate that a multimodal activity recognition approach can achieve more accurate classifications than single-sensor approaches. In the second experiment, a person interferes with the forward movement of the robot by pulling its base backward. This happens in a way that is not detected by Pepper’s inbuilt systems. The approach can detect the unexpected behavior and could be used to extend Pepper’s inbuilt capabilities. Through its generality, this work can be used to recognize activities of other robots with comparable sensing capabilities.