Detection of upper limb abrupt gestures for human–machine interaction using deep learning techniques

Abstract

In the manufacturing industry the productivity is contingent on the workers’ well-being, with operators at the center of the production process. Moreover, when human–machine interaction occurs, operators’ safety is a key requirement. Generally, typical human gestures in manipulation tasks have repetitive kinetics, however external disturbances or environmental factors might provoke abrupt gestures, leading to improper interaction with the machine. The identification and characterization of these abrupt events has not yet been thoroughly studied. Accordingly, the aim of the current research was to define a methodology to ready identify human abrupt movements in a workplace, where manipulation activities are carried out. Five subjects performed three times a set of 30 standard pick-and-place tasks paced at 20 bpm, wearing magneto-inertial measurement units (MIMUs) on their wrists. Random visual and acoustic alarms triggered abrupt movements during standard gestures. The recorded signals were processed by segmenting each pick-and-place cycle. The distinction between standard and abrupt gestures was performed through a recurrent neural network applied to acceleration signals. Four different pre-classification methodologies were implemented to train the neural network and the resulting confusion matrices were compared. The outcomes showed that appropriate preprocessing of the data allows more effective training of the network and shorter classification time, enabling to achieve accuracy greater than 99% and F1-score better than 90%.