Hybrid multi-modal fusion for human action recognition

Abstract

We introduce in this paper a hybrid fusion approach allowing the efficient combination of the Kinect modalities within the feature, representation and decision levels. Our contributions are three-fold: (i) We propose an efficient concatenation of complementary per-modality descriptors that rely on the joint modality as a high-level information. (ii) We apply a multi-resolution analysis that combines the local frame-wise decisions with the global BoVW ones. We rely in this context on the scalability of the Fisher vector representation in order to handle large-scale data and apply additional concatenation of its output. (iii) We also propose an efficient score merging scheme by generating multiple weighting-coefficients that combine the strength of different SVM classifiers with a given action label. By evaluating our approach on the Cornell activity dataset, state-of-the-art performances are obtained.