Trace Anything: Representing Any Video in 4D via Trajectory Fields

TL;DR

Trajectory Fields introduce a dense 4D video representation that assigns every pixel in every frame a 3D trajectory over time. Trace Anything, a one-pass, feed-forward predictor, outputs per-pixel spline control points to assemble these trajectories without depth, flow, or per-scene optimization. A Blender-based data platform supports large-scale training and evaluation with dense 4D annotations, yielding state-of-the-art trajectory-field estimation and competitive point tracking, plus emergent abilities like forecasting and spatio-temporal fusion. The work demonstrates significant inference efficiency and broad applicability to dynamic scene understanding while outlining future directions to bridge real-world data gaps and enhance expressivity.

Abstract

Effective spatio-temporal representation is fundamental to modeling, understanding, and predicting dynamics in videos. The atomic unit of a video, the pixel, traces a continuous 3D trajectory over time, serving as the primitive element of dynamics. Based on this principle, we propose representing any video as a Trajectory Field: a dense mapping that assigns a continuous 3D trajectory function of time to each pixel in every frame. With this representation, we introduce Trace Anything, a neural network that predicts the entire trajectory field in a single feed-forward pass. Specifically, for each pixel in each frame, our model predicts a set of control points that parameterizes a trajectory (i.e., a B-spline), yielding its 3D position at arbitrary query time instants. We trained the Trace Anything model on large-scale 4D data, including data from our new platform, and our experiments demonstrate that: (i) Trace Anything achieves state-of-the-art performance on our new benchmark for trajectory field estimation and performs competitively on established point-tracking benchmarks; (ii) it offers significant efficiency gains thanks to its one-pass paradigm, without requiring iterative optimization or auxiliary estimators; and (iii) it exhibits emergent abilities, including goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion. Project page: https://trace-anything.github.io/.

Figures (17)

• Figure 1: Any video$^{*}$ can be represented in 4D with a Trajectory Field, a dense mapping assigning each pixel in each frame to a parametric 3D trajectory. We propose Trace Anything, a neural network that predicts the trajectory field with a single forward pass.
• Figure 2: Given the input frames (left), a trajectory field represents the video at the atomic level, mapping each pixel in each frame to a 3D trajectory, expressed as a parametric curve (right).
• Figure 3: Trace Anything pipeline. Input frames are processed by a geometric backbone consisting of an image encoder and a fusion transformer. The control point head outputs dense control point maps $\mathbf{P}_i \in \mathbb{R}^{D \times H \times W \times 3}$, where $\mathbf{P}^{(k)}_{i,u,v}$ is the $k$-th control point for pixel $(u,v)$ in frame $I_i$. These define continuous 3D trajectories $\mathbf{x}_{i,u,v}(t)$ via cubic B-splines, yielding a 4D reconstruction.
• Figure 4: Sample renderings from our data platform.
• Figure 5: Video-based trajectory field estimation on DAVIS perazzi2016benchmark. Trace Anything predicts trajectory fields that can yield dynamic point cloud sequences and dense 3D trajectories, while remaining robust to complex non-rigid motion and occlusions.