Weakly-supervised action recognition and localization via knowledge transfer

Abstract

Action recognition and localization has attracted much attention in the past decade. However, a challenging problem is that it typically requires large-scale temporal annotations of action instances for training models in untrimmed video scenarios, which is not practical in many real-world applications. To alleviate the problem, we propose a novel weakly-supervised action recognition framework for untrimmed videos to use only video-level annotations to transfer information from publicly available trimmed videos to assist in model learning, namely KTUntrimmedNet. A two-stage method is designed to guarantee an effective transfer strategy: Firstly, the trimmed and untrimmed videos are clustered to find similar classes between them, so as to avoid negative information transfer from trimmed data. Secondly, we design an invariant module to find common features between trimmed videos and untrimmed videos for improving the performance. Extensive experiments on the standard benchmark datasets, THUMOS14 and ActivityNet1.3, clearly demonstrate the efficacy of our proposed method when compared with the existing state-of-the-arts.