When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning

TL;DR

This work tackles self-supervised video representation learning by addressing two key challenges in Masked Video Modeling: reconstruction uncertainty from random temporal sampling and excessive low-level detail from pixel-space restoration. It introduces T-CoRe, which combines a sandwich sampling strategy to fuse information from past and future frames with a latent-space Patch Matching Module inside a mean-teacher (self-distillation) framework to recover high-level semantic representations. The method jointly optimizes patch-level latent reconstruction and frame-level semantic alignment using a combination of L_pt, L_ft, L_pf, L_DINO, and L_koleo losses, outperforming state-of-the-art SSL approaches on DAVIS-2017, VIP, and JHMDB and showing scalability to larger backbones and ImageNet pretraining. Overall, T-CoRe demonstrates that leveraging temporal correspondence in latent space yields more compact, temporally coherent representations that improve downstream video understanding tasks and generalize across datasets.

Abstract

The past decade has witnessed notable achievements in self-supervised learning for video tasks. Recent efforts typically adopt the Masked Video Modeling (MVM) paradigm, leading to significant progress on multiple video tasks. However, two critical challenges remain: 1) Without human annotations, the random temporal sampling introduces uncertainty, increasing the difficulty of model training. 2) Previous MVM methods primarily recover the masked patches in the pixel space, leading to insufficient information compression for downstream tasks. To address these challenges jointly, we propose a self-supervised framework that leverages Temporal Correspondence for video Representation learning (T-CoRe). For challenge 1), we propose a sandwich sampling strategy that selects two auxiliary frames to reduce reconstruction uncertainty in a two-side-squeezing manner. Addressing challenge 2), we introduce an auxiliary branch into a self-distillation architecture to restore representations in the latent space, generating high-level semantic representations enriched with temporal information. Experiments of T-CoRe consistently present superior performance across several downstream tasks, demonstrating its effectiveness for video representation learning. The code is available at https://github.com/yafeng19/T-CORE.

Figures (12)

• Figure 1: (a) MIM restores the masked image patches with the assistance of spatially adjacent patches, while MVM restores the masked frame patches by establishing correspondence from temporally adjacent frames. (b) Our framework outperforms state-of-the-art methods on multiple video downstream tasks such as video object segmentation on DAVIS-2017 DAVIS17.
• Figure 2: Top: inferring the object in the current frame from a random past frame suffers from high uncertainty. Bottom: our sandwich sampling reduces the uncertainty in a two-side-squeezing manner, such that the current object can be precisely located.
• Figure 3: The architecture of our proposed framework. Using the sandwich sampling technique, we sample the past, current, and future frames from the given video. The current frame is fed into both the student and teacher branches, with a random mask applied to the student branch. The past and future frames are processed by an auxiliary branch to provide temporal correspondence through the Patch Matching Module. The reconstructed representations are then aligned with the teacher branch and pulled closer to reduce uncertainty.
• Figure 4: Structure of the Patch Matching Module. With the auxiliary of the past frame, $\bm{z}_c^p = g(\bm{z}_p, \bar{\bm z}_c)$ is generated via three blocks in PMM. The process is the same for the future frame.
• Figure 5: Visualization results of T-CoRe for human pose propagation on JHMDB (top) and body part propagation on VIP (bottom).