Recognizing human actions by sharing knowledge in implicit action groups

Abstract

Most of the current action recognition approaches learn each action category separately. An important observation is that many action categories are correlated and could be clustered into groups, which are always ignored to decreasing the recognition accuracy. In this paper, we employ a multi-task learning framework with group-structured regularization to share knowledge in category groups. First, we employ Fisher Vector, concatenated by gradients with respect to mean vector and covariance matrix of GMM, to represent action data. Intuitively, the action categories in the same group are prone to have a closer relationship with the same Gaussian components. The proposed method uses one-vs-one SVM margin to measure the degree of similarity between each pair of categories and obtain the implicit group structure by Affinity Propagation Clustering. In order to encourage the categories in the same group to share dimensions feature from the same Gaussian component and vice versa, the implicit group structure is used as the prior regularization in multi-task learning. Our experiments on large and realistic dataset HMDB51 show that the proposed method has achieved the comparative even higher accuracy with less dimensions of feature over several state-of-the-art approaches.