TAPVid-3D: A Benchmark for Tracking Any Point in 3D

TL;DR

TAPVid-3D addresses the lack of real-world 3D long-range point-tracking benchmarks by unifying three diverse data sources and providing ground-truth 3D trajectories with occlusion labels. It introduces TAP-3D-specific metrics, including $APD_{3D}$, $OA$, and $3D$-$AJ$, combined with scalable depth-rescaling strategies to accommodate different applications. The benchmark offers three data sources (Aria Digital Twin, DriveTrack, Panoptic Studio) and clear evaluation splits, along with baseline analyses that reveal the gap between 2D TAP performance and true 3D tracking accuracy. By enabling standardized evaluation of monocular video for 3D motion understanding, TAPVid-3D aims to accelerate development of models capable of robust, occlusion-aware dynamic scene understanding and downstream robotics tasks.

Abstract

We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io

Figures (12)

• Figure 1: Random samples from TAPVid-3D: on the top row, we visualize the point trajectories projected into the 2D video frame; on the bottom row, we visualize the metric 3D point trajectories. From left to right, we show one example from each constituent data source: ADT, DriveTrack and Panoptic Studio.
• Figure 2: Illustrative TAP-3D results of BootsTAPIR + ZoeDepth. We compare the ground-truth 2D and 3D tracks (blue solid) with the predicted tracks (red dotted). (a) Accurate tracking. (b) Noisy depth estimations result in a noisy 3D track. (c) Inconsistent depth scales across time (scale drift). (d) Inconsistent depth scales across space don't allow a single global scale factor to properly fit all tracks.
• Figure 3: Statistics on TAPVid-3D. Top left: video lengths. Top right: Number of trajectories in each clip. Bottom left: Number of static tracks in each clip. Bottom right: average point velocity.
• Figure 4: Random samples from ADT subset in TAPVid-3D: on the top row, we visualize the point trajectories projected into the 2D video frame; on the bottom row, we visualize the metric 3D point trajectories. For each video, we show 3 frames sampled at time step 30, 60 and 90.
• Figure 5: Random samples from ADT subset in TAPVid-3D (cont'd.): on the top row, we visualize the point trajectories projected into the 2D video frame; on the bottom row, we visualize the metric 3D point trajectories. For each video, we show 3 frames sampled at time step 30, 60 and 90.