Tracking by Predicting 3-D Gaussians Over Time

TL;DR

This work tackles learning robust pixel-wise correspondence in video without annotations. It introduces Video-GMAE, which encodes frames as moving 3-D Gaussian primitives and trains with a masked autoencoder objective that also predicts temporal deltas, leveraging differentiable Gaussian rendering. The approach yields zero-shot point tracking that rivals self-supervised baselines and, with finetuning, achieves strong gains on multiple datasets, outperforming existing video SSL methods. By tying video representations to a 3-D Gaussian decomposition, the method emphasizes temporal structure and scene motion, offering practical benefits for scalable video understanding.

Abstract

We propose Video Gaussian Masked Autoencoders (Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats moving over time. Representing a video as a set of Gaussians enforces a reasonable inductive bias: that 2-D videos are often consistent projections of a dynamic 3-D scene. We find that tracking emerges when pretraining a network with this architecture. Mapping the trajectory of the learnt Gaussians onto the image plane gives zero-shot tracking performance comparable to state-of-the-art. With small-scale finetuning, our models achieve 34.6% improvement on Kinetics, and 13.1% on Kubric datasets, surpassing existing self-supervised video approaches. The project page and code are publicly available at https://videogmae.org/ and https://github.com/tekotan/video-gmae.

Figures (16)

• Figure 1: Self-supervised Video Pretraining for Correspondence: Given a sequence of video frames, our approach Video-GMAE predicts Gaussian primitives for each frame to reconstruct the whole video. In addition to this, we also enforce correspondence in the Gaussian primitives by predicting the delta Gaussians for all but the first frame.
• Figure 2: Video Masked Auto encoding via Gaussian Splatting: The ViT Encoder processes masked input frames to produce latent embeddings. The ViT Decoder then predicts explicit Gaussian parameters for frame $f_1$ based on query tokens, including color, opacity, center, scale, and orientation, and the Gaussian deltas for the rest of the frames. The explicit Gaussians for frame $f_1$ to $f_t$ are calculated and rendered via differentiable volume splitting to reconstruct all the frames. We pre-train our models fully end-to-end with self-supervision.
• Figure 3: Zero-shot Point Tracking: The 3-D centers of the predicted Gaussian primitives are projected and the subsequent 2-D displacement vector is rendered.
• Figure 4: Finetuning for Point Tracking: We use our pretrained encoder without masking, and query the latents to predict the best point tracks. We finetune this model, using the annotated Kubric greff2022kubric dataset. The Fourier embeddings of the initial queries are calculated, and they cross-attend to the encoder latents in the fine-tuned cross-attention readout layer.
• Figure 5: Zero shot Qualitative Results: We show qualitative results (16 frames, we only show every other frame here) from our Video-GMAE-zeroshot method. Without any track labels, the model can track objects with camera motion and pose changes, and shows robust tracking over long videos.