Learning complex spatio-temporal configurations of body joints for online activity recognition

Abstract

Geometric dynamic configurations of body joints play an essential role in distinguishing different human activities. However, many existing human activity recognition approaches lack the capability of automatically learning these configurations from sequences of joints in four-dimensional space (spatio and temporal). In this paper, the authors propose an automatic joint configuration learning method, based on dictionary learning and sparse representation. The proposed method achieves the following features: 1) it automatically learns dynamic spatio-temporal geometric configurations of body joints, involved in activities, in a simple way; 2) it dispenses with the hand crafted feature designing process and provides a new method to organize joint coordinate data as fixed length column vectors, which are suitable for dictionary learning; 3) it replaces the conventional bag of words model with sparse coding method; words in learned dictionary capture subactivity features, and the frequencies of different words appearing in different activities characterize the categories of global activity; 4) it is robust to time misalignment and can classify any length of video sequence (online classification) in real time; 5) it is easy to combine this method with other forms of data for better performance, because of its data driven nature and flexible framework. The proposed method is tested with three state-of-the-art public human activity recognition datasets and the results are found to be better than those of CAD-60 dataset, and comparable to those of both MSR Action 3D and MSR Daily Activity datasets (source codes are publicly available at https://github.com/jinqijinqi/SparseCodingDictionaryLearningHumanActivityRecognition).