Structure-Feature Fusion Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition

Abstract

Human skeleton contains intuitive information of actions and has high robustness in dynamic environment. Therefore, it has been widely studied in action recognition tasks. Most existing methods of skeleton recognition are based on graph convolutional networks (GCNs), which extract the topological structure of graphs to describe the dependencies between joints. However, the GCNs pay excessive attention to skeleton structure and neglect the feature information of skeleton joints. Accordingly, how to fuse the feature of both skeleton structure and joints is a problem to be solved. In addition, non-linear temporal convolutional network (TCN), which has more robustness and learning capability, is rarely investigated in existing methods. With the comprehensive consideration of the dependence between structure and feature on graphs, we propose a novel structure-feature fusion adaptive GCN (SFAGCN) for skeleton based action recognition. The topological structure of the skeleton graph and the feature of the joints can be fused by the decoupled spatiotemporal correlation in our model effectively. The relevance of spatiotemporal data can be preserved well by the fusion strategy, with data integrity ensured. Moreover, Gated TCN is used to extract temporal feature, improving the network performance further. We choose two-stream adaptive GCNs and shift-GCN as the baseline. To demonstrate the effectiveness of our methods, extensive experiments are implemented on the three large-scale datasets, namely, NTU-RGBD 60, NTU-RGBD 120 and Kinetics-Skeleton 400. The accuracy of top-1 on above datasets are improved by more than 0.6% on average, where the performance of SFAGCN exceeds the state-of-the-art methods.