Generative Point Tracking with Flow Matching

TL;DR

Generative Point Tracker is capable of capturing the multi-modality in point trajectories, which translates to state-of-the-art tracking accuracy on occluded points, while maintaining competitive tracking accuracy on visible points compared to extant discriminative point trackers.

Abstract

Tracking a point through a video can be a challenging task due to uncertainty arising from visual obfuscations, such as appearance changes and occlusions. Although current state-of-the-art discriminative models excel in regressing long-term point trajectory estimates -- even through occlusions -- they are limited to regressing to a mean (or mode) in the presence of uncertainty, and fail to capture multi-modality. To overcome this limitation, we introduce Generative Point Tracker (GenPT), a generative framework for modelling multi-modal trajectories. GenPT is trained with a novel flow matching formulation that combines the iterative refinement of discriminative trackers, a window-dependent prior for cross-window consistency, and a variance schedule tuned specifically for point coordinates. We show how our model's generative capabilities can be leveraged to improve point trajectory estimates by utilizing a best-first search strategy on generated samples during inference, guided by the model's own confidence of its predictions. Empirically, we evaluate GenPT against the current state of the art on the standard PointOdyssey, Dynamic Replica, and TAP-Vid benchmarks. Further, we introduce a TAP-Vid variant with additional occlusions to assess occluded point tracking performance and highlight our model's ability to capture multi-modality. GenPT is capable of capturing the multi-modality in point trajectories, which translates to state-of-the-art tracking accuracy on occluded points, while maintaining competitive tracking accuracy on visible points compared to extant discriminative point trackers.

Figures (10)

• Figure 1: Multi-modal prediction of point trajectories. (Top) 100 randomly sampled trajectories of a single tracked point from our model, Generative Point Tracker (GenPT). The first image shows the full path of each sampled trajectory. Our model captures the multi-modality present in uncertain regions of the video. When tracking uncertainty is high, the majority of predictions are clustered around regions where the point is most likely to be. Occasionally, spurious (but valid) predictions can occur due to feature similarity elsewhere in the frame. Video is from the TAP-Vid RGB-S dataset doersch2022tap. (Middle) Ground truth for top row. (Bottom) Point trajectory estimates can be improved by utilizing a best-first search strategy on generated samples during inference, guided by the model's own confidence of its predictions in each window of observation.
• Figure 2: Architecture overview. GenPT is a generative point tracker based on a modified flow matching setup. (i) Given a query point, $P^q$, to track in a given video, GenPT starts by computing features for each frame of the video using a convnet, $F_t = \mathcal{F}_\phi(I_t)$, and sampling a neighbourhood of features around $P^q$. (ii) GenPT generates samples in a sliding window (i.e. causal) manner, where samples for each window, $\{ \tilde{P}_l \}_{l=1}^L$, are composited sequentially. If in the first window, the initial sample $\tilde{P}_1$ is sampled from a Gaussian centred about $P^q$, otherwise it is initialized at sample $\tilde{P}_L$ from the previous window, with Gaussian noise added at non-overlapping frames. (iii) From sample $\tilde{P}_l$, a transformer $\mathcal{T}_\theta$ is used to estimate the ground truth trajectory $P$, taking $K$ refinement steps. Each step is guided by correlation features that are computed using query features and features centred about the current estimate, $\hat{P}_k$. (iv) The final estimate, $\hat{P}_K$, is used to integrate to the next sample, $\tilde{P}_{l+1}$. For simplicity, the figure excludes visibilities and confidences.
• Figure 3: Multi-modal prediction due to spatial homogeneity (left) and self-occlusion (right). (Left) When the tracked point is a feature that is spatially homogeneous, predictions become increasingly varied as time moves on, especially in the presence of occlusion (top-left). As expected, these predictions cluster around neighbouring visually similar areas. Videos shown are from the TAP-Vid RGB-S dataset doersch2022tap. (Right) As the subject rotates, the tracked point becomes increasingly occluded, resulting in increasingly varied predictions that cluster around the expected location of the point. Videos shown are from the TAP-Vid DAVIS dataset doersch2022tap.
• Figure 4: Target reacquisition post-occlusion. When the point being tracked is uniquely identifiable in the scene, predictions made behind an occluder cluster around the expected location of the point. However, when the occluder passes, the variance of predictions rapidly shrinks. Videos shown are from the TAP-Vid DAVIS dataset doersch2022tap.
• Figure 5: Transformer architecture overview. GenPT uses a transformer, $\mathcal{T}_\theta$, that processes point trajectory, visibility, and confidence inputs as spatiotemporal tokens, using factorized temporal and spatial attention blocks, similarly to CoTracker karaev2024cotracker. disp contains the forward and backward per-frame coordinate displacement computed from $\hat{P}_{n_T,k,l}$.