TrackMAE: Video Representation Learning via Track Mask and Predict

Abstract

Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos, generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction spaces by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines, learning more discriminative and generalizable representations. Code available at https://github.com/rvandeghen/TrackMAE

Figures (6)

• Figure 1: TrackMAE improves masked video modeling by jointly predicting spatial features and motion trajectories in a mask-and-predict fashion.
• Figure 2: Overview of TrackMAE. In the lower branch, a video clip $\mathbf{V}$ is first patchified and masked. The visible tokens are fed to a ViT encoder $\Phi$. Then the decoder $\Psi_{spatial}$ aims to reconstruct spatial features based on the encoder output. In the upper branch, the input video clip is processed by a CoTracker3 module, extracting sparse point trajectories. The encoder output is then passed to a second decoder $\Psi_{Motion}$, which aims to predict the extracted trajectories. The training objective combines both motion and spatial reconstruction.
• Figure 3: Masking comparison. We show how our motion-based tube masking compares to random tube masking. By explicitly sampling visible tokens, our motion-based sampling distribution ensures that visible tokens cover both motion and static regions. In the motion-based tube sampling, red squares are sampled from the high-motion and blue squares from the low-motion bin.
• Figure 4: Training evolution. We report the Top-1 Accuracy for K400 and SSv2 finetuning at different pretraining epochs. Our model trained with both CLIP and trajectory reconstructions consistently outperforms the CLIP reconstruction only.
• Figure 5: Sampling strategies. We show how we can use the motion information to create different sampling distribution.