Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is This Tracker On? A Benchmark Protocol for Dynamic Tracking

Ilona Demler, Saumya Chauhan, Georgia Gkioxari

TL;DR

ITTO confronts the gap between real-world dynamic tracking and existing benchmarks by providing a long-range, real-world dataset with substantially higher motion complexity, occlusions, and reappearances. The benchmark employs a two-stage, bias-free annotation pipeline and algorithmic query-point sampling to yield high-quality, diverse tracks, enabling diagnostic analysis across axes of motion and reappearance, including a new PDV metric for spatial coherence. Across ten SOTA trackers, ITTO reveals dramatic performance drops relative to TAP-Vid, highlighting failure modes such as re-identification after occlusion, memory limitations, and sensitivity to motion severity, even for 3D methods. The findings advocate memory-rich, coarse-to-fine, or object-pointer-inspired architectures and position ITTO as a foundational testbed for advancing robust, real-world point tracking, with potential extensions to 3D and broader domains.

Abstract

We introduce ITTO, a challenging new benchmark suite for evaluating and diagnosing the capabilities and limitations of point tracking methods. Our videos are sourced from existing datasets and egocentric real-world recordings, with high-quality human annotations collected through a multi-stage pipeline. ITTO captures the motion complexity, occlusion patterns, and object diversity characteristic of real-world scenes -- factors that are largely absent in current benchmarks. We conduct a rigorous analysis of state-of-the-art tracking methods on ITTO, breaking down performance along key axes of motion complexity. Our findings reveal that existing trackers struggle with these challenges, particularly in re-identifying points after occlusion, highlighting critical failure modes. These results point to the need for new modeling approaches tailored to real-world dynamics. We envision ITTO as a foundation testbed for advancing point tracking and guiding the development of more robust tracking algorithms.

Is This Tracker On? A Benchmark Protocol for Dynamic Tracking

TL;DR

ITTO confronts the gap between real-world dynamic tracking and existing benchmarks by providing a long-range, real-world dataset with substantially higher motion complexity, occlusions, and reappearances. The benchmark employs a two-stage, bias-free annotation pipeline and algorithmic query-point sampling to yield high-quality, diverse tracks, enabling diagnostic analysis across axes of motion and reappearance, including a new PDV metric for spatial coherence. Across ten SOTA trackers, ITTO reveals dramatic performance drops relative to TAP-Vid, highlighting failure modes such as re-identification after occlusion, memory limitations, and sensitivity to motion severity, even for 3D methods. The findings advocate memory-rich, coarse-to-fine, or object-pointer-inspired architectures and position ITTO as a foundational testbed for advancing robust, real-world point tracking, with potential extensions to 3D and broader domains.

Abstract

We introduce ITTO, a challenging new benchmark suite for evaluating and diagnosing the capabilities and limitations of point tracking methods. Our videos are sourced from existing datasets and egocentric real-world recordings, with high-quality human annotations collected through a multi-stage pipeline. ITTO captures the motion complexity, occlusion patterns, and object diversity characteristic of real-world scenes -- factors that are largely absent in current benchmarks. We conduct a rigorous analysis of state-of-the-art tracking methods on ITTO, breaking down performance along key axes of motion complexity. Our findings reveal that existing trackers struggle with these challenges, particularly in re-identifying points after occlusion, highlighting critical failure modes. These results point to the need for new modeling approaches tailored to real-world dynamics. We envision ITTO as a foundation testbed for advancing point tracking and guiding the development of more robust tracking algorithms.
Paper Structure (33 sections, 4 equations, 12 figures, 9 tables)

This paper contains 33 sections, 4 equations, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tracking methods.
  4. Point tracking benchmarks.
  5. Object tracking datasets.
  6. The ITTO Tracking Benchmark
  7. Comparison to existing tracking benchmarks
  8. Benchmark collection & annotation protocol
  9. Annotation pipeline.
  10. Ease of tool use.
  11. Query point selection.
  12. Validation of annotation quality.
  13. Ethics of annotation collection
  14. Evaluation and Metrics
  15. Point tracking methods.
  16. ...and 18 more sections

Figures (12)

  • Figure 1: Overview of ITTO benchmark vs. other benchmarks. ITTO has more objects per video, highly deformable objects (such as umbrella opening and closing in top row) and all phenomena of the real world (significant % of moving tracks, and high track reappearance count). SOTA tracking methods perform worse on ITTO compared to the existing benchmark; e.g. CoTracker3 cotracker3 performance drops from 61.8 to 35.5 for Average Jaccard (AJ) and from 76.1 to 47.1 for $\delta$.
  • Figure 2: CoTracker3 on ITTO. Points tracked by color & red boxes added for illustration. Top: model biases red umbrella tracks to the right panel and fails when the girls switch positions. Middle: model struggles to track individual monkeys, which have repeated features. Bottom: model fails under camera-to-object distance changes. Background points are tracked with near-perfect accuracy.
  • Figure 3: Catastrophic Track Failure. Track failure rates over time for 2, 4, and 6 pixel thresholds. CoTracker3 and DELTA suffer rapid and irrecoverable degradation once tracks leave their native resolution window. LocoTrack is more resilient, likely due to its global track initialization stage.
  • Figure 4: Annotation Pipeline. Left hand side shows Amazon Mechanical Turk setup. MTurk workers are given a reference image and a query image side-by-side, and place visible keypoints on query images for which the point is visible. If the point is occluded, they select "label as hidden'" for that frame. Right-hand side shows the video refinement tool. Annotators play through the video and correct the point location and visibility flag through the arrows on the control panel. Alternatively they can also use keyboard controls to play through the video and refine the tracks. Annotators play through the video and correct frames as many times as they need until the track is spatiotemporally consistent.
  • Figure 5: Pseudomasks. We generate pseudomasks that we generate by first prompting Molmo to generate three positive and two negative coordinates for input semantic concepts. We send these labels into SAM2, which generates pseudomasks. For the leftmost video, we prompt with "ball", "frame", "net", and "hand"; for the middle video we prompt with "engine", "tire", "hand" "foot", and "pant"; and for the rightmost video we prompt with "head", "GoPro", "hand", and "forearm".
  • ...and 7 more figures