Point Prompting: Counterfactual Tracking with Video Diffusion Models

TL;DR

Through experiments with multiple image-conditioned video diffusion models, it is found that these "emergent"tracks outperform those of prior zero-shot methods and persist through occlusions, often obtaining performance that is competitive with specialized self-supervised models.

Abstract

Trackers and video generators solve closely related problems: the former analyze motion, while the latter synthesize it. We show that this connection enables pretrained video diffusion models to perform zero-shot point tracking by simply prompting them to visually mark points as they move over time. We place a distinctively colored marker at the query point, then regenerate the rest of the video from an intermediate noise level. This propagates the marker across frames, tracing the point's trajectory. To ensure that the marker remains visible in this counterfactual generation, despite such markers being unlikely in natural videos, we use the unedited initial frame as a negative prompt. Through experiments with multiple image-conditioned video diffusion models, we find that these "emergent" tracks outperform those of prior zero-shot methods and persist through occlusions, often obtaining performance that is competitive with specialized self-supervised models.

Figures (10)

• Figure 1: Prompting a diffusion model for tracking. (a) We use an off-the-shelf video diffusion model to perform point tracking. We add a small, distinctive marking---a red dot---to the first frame of an input video, then ask the diffusion model to regenerate the rest of the video using SDEdit meng2021sdedit, which propagates the marking to subsequent frames. (b) We then track the motion of this marking over time. This motion corresponds to the trajectory of the underlying physical point. The model successfully tracks through occlusion. Please see the webpage for more results: https://point-prompting.github.io.
• Figure 2: Enhancing the Counterfactual Signal. We use negative prompting to ensure that the generated video contains the marker. In each denoising step (Eq. \ref{['eq:guidance_ddpm']}), we condition the denoising on two images: (1) Edited First Frame: the first frame of the video with a marking added, and (2) Unedited First Frame: the original first frame of the video. We then subtract the weighted noise vector of the latter from the former.
• Figure 2: Video Model Ablations. Wan2.1-1.3B and 14B wang2025wan outperform CogVideoX Yang2024CogVideoXTD, showing that stronger video models improve tracking performance.
• Figure 3: Tracking Enhancements. To improve point tracking in video, we introduce two enhancements: (1) Color Rebalancing: remove existing red hues to ensure the red marker remains a unique tracking cue; (2) Refinement: obtain initial trajectories with a color-based tracker, then refine them using an inpainting mask to correct temporal artifacts such as object shifts (as shown in white circles). This two-step procedure first produces coarse tracks and then refines them via mask-constrained reverse diffusion.
• Figure 4: Tracking Pipeline Ablations. Quantitative results on TAP-Vid DAVIS-First showing the impact of each stage in our pipeline (Fig. \ref{['fig:pipeline']}). The last row uses original pixel color instead of the red dot for tracking.