A Novel Scheme for Training Two-Stream CNNs for Action Recognition

Abstract

Human actions recognition from realistic video data constitutes a challenging and relevant research area. Leading the state-of-the-art we can find those methods based on Convolutional Neural Networks (CNNs) and specially two-stream CNNs (appearance and motion). In this paper we present a novel scheme for training two-stream CNNs that increases the accuracy of the fusion (when one of the channels does not perform as well as the other one) and reduces the total time used for training the entire architecture. In addition, we introduce a new descriptor for motion representation that improves the state-of-the-art. Based on this more efficient scheme, we developed an early recognition system. The proposed approach is evaluated on the UCF101 data set with competitive results.