Egocentric Hand Gesture Recognition on Untrimmed Videos Using State Activation Gate LSTMs

Abstract

Deep Neural Networks have been used for recognising ego-hand gestures in trimmed videos extensively. However, recognising ego-hand gestures from untrimmed videos has been largely unexplored. In this work, we propose the concept of State Activation Gate (StAG) to extend the current LSTM framework and successfully apply it to recognising ego-hand gestures from untrimmed videos. We explore the usage of StAG LSTM combined with 3D convolutional neural networks to compare their performance to the state-of-the-art for two publicly available datasets. In addition, we present an intra-gesture (IG) loss function and a metric that favours continuity of gesture labels called Continuity Favouring Jaccard Index (CFJI). StAG LSTM reduces the need to use heuristics currently employed in ego-hand gesture recognition on untrimmed videos. Using the proposed IG loss function for training, achieves better performance on metrics like Jaccard Index (JI) and AUC scores compared to the state of the art.