UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

TL;DR

UnSAMv2 introduces a granularity-aware, self-supervised framework that enables segment-anything at any granularity without human annotations. By building dense mask–granularity pairs through a four-stage, divide-and-conquer pipeline and embedding a continuous granularity scalar into the SAM-2 decoder via a Fourier-based granularity encoder, the approach achieves continuous, parts-to-wholes segmentation while requiring only 6{,}000 unlabeled images and negligible parameter overhead. The method delivers state-of-the-art performance across interactive, whole-image, and video segmentation benchmarks and offers a light-supervised variant (UnSAMv2+) that further improves results with SA-1B labels. This work demonstrates that latent hierarchical structure and scale-aware segmentation can be learned unsupervised within a vision foundation model, enabling flexible, low-cost, and user-driven objectness concepts.

Abstract

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which enables segment anything at any granularity without human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only $6$K unlabeled images and $0.02\%$ additional parameters, UnSAMv2 substantially enhances SAM-2, achieving segment anything at any granularity across interactive, whole-image, and video segmentation tasks. Evaluated on over $11$ benchmarks, UnSAMv2 improves $\text{NoC}_{90}$ (5.69 $\rightarrow$ 4.75), 1-IoU (58.0 $\rightarrow$ 73.1), and $\text{AR}_{1000}$ (49.6 $\rightarrow$ 68.3), showing that small amounts of unlabeled data with a granularity-aware self-supervised learning method can unlock the potential of vision foundation models.

Figures (13)

• Figure 1: UnSAMv2 achieves state-of-the-art performance across interactive segmentation benchmarks. Across multiple datasets, UnSAMv2 consistently outperforms SAM-2 and prior methods, turning segmentation into a controllable and interpretable process rather than a fixed prediction.
• Figure 2: From ambiguity to control. Without granularity input, SAM-2 yields up to three masks per point, requiring users to manually choose one. UnSAMv2 resolves this ambiguity by introducing a continuous granularity variable, allowing users to obtain the intended object at any scale with a single prompt. This simple addition turns segmentation from a discrete guess into a continuous, controllable reasoning process.
• Figure 3: Granularity distribution of discovered masks. Our divide-and-conquer pipeline produces a rich, left-tailed hierarchy of pseudo-masks, dominated by fine-grained structures. Despite this imbalance, UnSAMv2 learns stable semantics across all scales. Hierarchical perception can emerge from unlabeled data!
• Figure 4: Granularity as a relative notion. At a fixed granularity value, mask sizes vary widely across scenes, showing that UnSAMv2 learns granularity relationally, consistent with human perception of parts and wholes rather than simply associating it with absolute size.
• Figure 5: Architecture of UnSAMv2. Built on SAM-2, UnSAMv2 introduces a Fourier-based granularity encoder and a granularity-aware mask token to enable segmentation at arbitrary granularity. A scalar granularity input $g\!\in\![0.1,1]$ is mapped to a high-dimensional embedding via Fourier transformation and an MLP, then injected into the transformer alongside the sparse point prompt embedding and dense image embedding. The granularity-aware mask token attends to image, point, and granularity embeddings, and is finally decoded by a token decoder into a mask at the requested granularity.