Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAM 2++: Tracking Anything at Any Granularity

Jiaming Zhang, Cheng Liang, Yichun Yang, Chenkai Zeng, Yutao Cui, Xinwen Zhang, Xin Zhou, Kai Ma, Gangshan Wu, Limin Wang

TL;DR

SAM 2++ unifies video tracking across granularities by using task-specific prompts for inputs, a unified decoding form for outputs, and a task-adaptive memory that decouples memory components per task. It extends SAM 2 with a data engine and Tracking-Any-Granularity (TAG) dataset to train and benchmark mask-, box-, and point-based tracking in a single framework. Empirical results show state-of-the-art performance across VOS, SOT, and PT benchmarks, improved generalization through multi-task memory and diverse data, and clear visualization of memory adaptations. The work delivers a scalable, multi-granularity tracking foundation and a valuable dataset for future research and benchmarking in unified video tracking.

Abstract

Video tracking aims at finding the specific target in subsequent frames given its initial state. Due to the varying granularity of target states across different tasks, most existing trackers are tailored to a single task and heavily rely on custom-designed modules within the individual task, which limits their generalization and leads to redundancy in both model design and parameters. To unify video tracking tasks, we present SAM 2++, a unified model towards tracking at any granularity, including masks, boxes, and points. First, to extend target granularity, we design task-specific prompts to encode various task inputs into general prompt embeddings, and a unified decoder to unify diverse task results into a unified form pre-output. Next, to satisfy memory matching, the core operation of tracking, we introduce a task-adaptive memory mechanism that unifies memory across different granularities. Finally, we introduce a customized data engine to support tracking training at any granularity, producing a large and diverse video tracking dataset with rich annotations at three granularities, termed Tracking-Any-Granularity, which represents a comprehensive resource for training and benchmarking on unified tracking. Comprehensive experiments on multiple benchmarks confirm that SAM 2++ sets a new state of the art across diverse tracking tasks at different granularities, establishing a unified and robust tracking framework.

SAM 2++: Tracking Anything at Any Granularity

TL;DR

SAM 2++ unifies video tracking across granularities by using task-specific prompts for inputs, a unified decoding form for outputs, and a task-adaptive memory that decouples memory components per task. It extends SAM 2 with a data engine and Tracking-Any-Granularity (TAG) dataset to train and benchmark mask-, box-, and point-based tracking in a single framework. Empirical results show state-of-the-art performance across VOS, SOT, and PT benchmarks, improved generalization through multi-task memory and diverse data, and clear visualization of memory adaptations. The work delivers a scalable, multi-granularity tracking foundation and a valuable dataset for future research and benchmarking in unified video tracking.

Abstract

Video tracking aims at finding the specific target in subsequent frames given its initial state. Due to the varying granularity of target states across different tasks, most existing trackers are tailored to a single task and heavily rely on custom-designed modules within the individual task, which limits their generalization and leads to redundancy in both model design and parameters. To unify video tracking tasks, we present SAM 2++, a unified model towards tracking at any granularity, including masks, boxes, and points. First, to extend target granularity, we design task-specific prompts to encode various task inputs into general prompt embeddings, and a unified decoder to unify diverse task results into a unified form pre-output. Next, to satisfy memory matching, the core operation of tracking, we introduce a task-adaptive memory mechanism that unifies memory across different granularities. Finally, we introduce a customized data engine to support tracking training at any granularity, producing a large and diverse video tracking dataset with rich annotations at three granularities, termed Tracking-Any-Granularity, which represents a comprehensive resource for training and benchmarking on unified tracking. Comprehensive experiments on multiple benchmarks confirm that SAM 2++ sets a new state of the art across diverse tracking tasks at different granularities, establishing a unified and robust tracking framework.
Paper Structure (29 sections, 7 equations, 9 figures, 16 tables)

This paper contains 29 sections, 7 equations, 9 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries: Segment Anything Model 2
  4. Model
  5. Unified Task Input and Output Processing
  6. Task-adaptive Memory
  7. Training and Inference Details
  8. Data
  9. Annotation Pipeline
  10. Data Engine
  11. Tracking-Any-Granularity Dataset
  12. Experiments
  13. Comparison to state-of-the-art on three tasks
  14. Exploration Studies
  15. Visualization
  16. ...and 14 more sections

Figures (9)

  • Figure 1: The overall of SAM 2++, including (a) tracking any granularity task, (b) our unified tracking foundation model, and (c) our Tracking-Any-Granularity dataset collected through our data engine. SAM 2++ is capable of tracking targets at any granularity.
  • Figure 2: The SAM 2++ architecture. When a new frame is received, the result is conditioned on the new prompt and/or stored memories. The initial target state at any granularity is converted into task-specific prompts for unified input. The Unified decoder predicts the task result for the current frame in unified mask form. Finally, the task-adaptive memory transforms diverse target states into unified memory.
  • Figure 3: Annotation pipeline of our Tracking-Any-Granularity dataset.
  • Figure 4: Examples of Tracking-Any-Granularity Dataset.
  • Figure 5: Statistics on sources and attributes distribution of Tracking-Any-Granularity Dataset. The link in (c) reflects the frequent co-occurrence of multiple attributes in a sequence.
  • ...and 4 more figures