Repurposing Video Diffusion Transformers for Robust Point Tracking

TL;DR

This work investigates using pre-trained video Diffusion Transformer (DiT) features as robust backbones for point tracking in real-world videos. It analyzes the temporal coherence and robustness benefits of DiTs over traditional ResNet-based backbones, attributes improved matching to full 3D spatio-temporal attention, and introduces DiTracker, which combines query-key attention, lightweight LoRA adaptation, and cost fusion with ResNet. DiTracker achieves state-of-the-art performance on ITTO-MOSE with significantly fewer training resources and matches or surpasses leading TAP-Vid results, while showing strong robustness to motion blur and occlusions. The results validate video DiT features as an effective and efficient foundation for robust point tracking in practical settings.

Abstract

Point tracking aims to localize corresponding points across video frames, serving as a fundamental task for 4D reconstruction, robotics, and video editing. Existing methods commonly rely on shallow convolutional backbones such as ResNet that process frames independently, lacking temporal coherence and producing unreliable matching costs under challenging conditions. Through systematic analysis, we find that video Diffusion Transformers (DiTs), pre-trained on large-scale real-world videos with spatio-temporal attention, inherently exhibit strong point tracking capability and robustly handle dynamic motions and frequent occlusions. We propose DiTracker, which adapts video DiTs through: (1) query-key attention matching, (2) lightweight LoRA tuning, and (3) cost fusion with a ResNet backbone. Despite training with 8 times smaller batch size, DiTracker achieves state-of-the-art performance on challenging ITTO benchmark and matches or outperforms state-of-the-art models on TAP-Vid benchmarks. Our work validates video DiT features as an effective and efficient foundation for point tracking.

Figures (7)

• Figure 1: Enhanced robustness and training efficiency of our DiT-based point tracking framework.DiTracker leverages pre-trained video Diffusion Transformer (DiT) features to outperform state-of-the-art methods such as CoTracker3 karaev2024cotracker3 in challenging real-world scenarios involving complex motion and frequent occlusions, while achieving comparable final performance with $\mathbf{10\times}$ faster convergence, substantially reducing training cost. These results demonstrate that pre-trained video DiT features constitute an effective and efficient foundation for robust point tracking.
• Figure 2: Overall Architecture of DiTracker. For long video sequences, input frames are divided into $N$ chunks with the global first frame prepended. Individual video frames are then encoded via VAE. These encoded latents are processed by a video DiT to extract query feature $q_i$ and key feature $k_j$, which are then used to compute a hierarchical local 4D matching cost $\mathcal{C}^{\text{DiT}}_{i,j}$. The video DiT local cost is subsequently fused with the ResNet local cost $\mathcal{C}^{\text{ResNet}}_{i,j}$. Finally, a tracking head refines the trajectories over $T$ iterations, updating displacement ($\Delta P$), visibility ($\Delta V$), and confidence ($\Delta C$).
• Figure 3: Qualitative results on ITTO-MOSE demler2025tracker benchmark. Our DiTracker predicts more accurate point trajectories under challenging real-world scenarios, including dynamic motions and occlusions, even surpassing CoTracker3 karaev2024cotracker3.
• Figure 4: Attention head analysis in 17th layer of CogVideoX-2B.
• Figure 5: Qualitative results on ITTO-MOSE demler2025tracker benchmark. Our DiTracker predicts more smooth and accurate point trajectories under challenging real-world scenarios, including large displacement, motion blur, and occlusions, even surpassing CoTracker3 karaev2024cotracker3.