Neural networks with Hidden Markov models in skeleton-based gesture recognition

Abstract

In Gesture Recognition (GR) tasks, a system with a traditional use of Hidden Markov Models (HMMs) usually serves as a baseline. Their performance is often not so good and therefore somehow overlooked. However, in recent years, especially in Automatic Speech Recognition (ASR), there are advanced methods proposed for this type of model which have been shown to improve significantly recognition results. Among them, the use of Neural Networks (NNs) instead of Gaussian Mixture Models (GMMs) for estimating emission probabilities of HMMs has been considered as one of biggest advances [1, 2, 3]. This fact implies that the performance of HMM-based models on GR need to be revised. For this reason, in this study, we show that by carefully tailoring NNs to a traditional HMM-based GR system, we can improve significantly the performance, hence, achieving very competitive results on a skeleton-based GR task which is defined by using Microsoft Research Cambridge 12 (MSRC-12) data [4]. It should be pointed out that, it is straightforward to apply our proposed techniques to more complicated GR tasks such as Sign Language Recognition [5], where basically a sequence of sign gestures need to be transcribed.