Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw

16Citations
Citations of this article
23Readers
Mendeley users who have this article in their library.

Abstract

This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a representation learned by detecting spatiotemporal continuity/discontinuity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task is to construct spatiotemporal (3D) jigsaw puzzles and learn to solve them. However, as we demonstrate in the experiments, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) whereby the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist. This provides sufficient cues for the model to reason about the continuity. Instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable. The four tasks aim to learn representations sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that our CSJ achieves state-of-the-art on various benchmarks.

Cite

CITATION STYLE

APA

Huo, Y., Ding, M., Lu, H., Huang, Z., Tang, M., Lu, Z., & Xiang, T. (2021). Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw. In IJCAI International Joint Conference on Artificial Intelligence (pp. 751–757). International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2021/104

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free