Universal Organizer of SAM for Unsupervised Semantic Segmentation

TL;DR

The paper tackles the difficulty of obtaining precise, boundary-quality masks in unsupervised semantic segmentation. It introduces UO-SAM, a SAM-based universal organizer that combines a Local Region Optimizer (LRO) for boundary refinement and a Global Region Optimizer (GRO) for contextual, image-wide segmentation, all without additional training. LRO uses foreground feature extraction, location confidence maps, and largest connected components to generate accurate prompts, then employs Cascaded Refinement for sharper edges. GRO broadens the scope with grid-based sampling and a category voting mechanism to ensure complete object coverage and robust fusion of local and global predictions, resulting in state-of-the-art performance across ImageNet-S$_{50}$PASS, PASCAL VOC, and COCO-Stuff datasets.

Abstract

Unsupervised semantic segmentation (USS) aims to achieve high-quality segmentation without manual pixel-level annotations. Existing USS models provide coarse category classification for regions, but the results often have blurry and imprecise edges. Recently, a robust framework called the segment anything model (SAM) has been proven to deliver precise boundary object masks. Therefore, this paper proposes a universal organizer based on SAM, termed as UO-SAM, to enhance the mask quality of USS models. Specifically, using only the original image and the masks generated by the USS model, we extract visual features to obtain positional prompts for target objects. Then, we activate a local region optimizer that performs segmentation using SAM on a per-object basis. Finally, we employ a global region optimizer to incorporate global image information and refine the masks to obtain the final fine-grained masks. Compared to existing methods, our UO-SAM achieves state-of-the-art performance.

Figures (3)

• Figure 1: State-of-the-art comparison on three datasets (i.e., ImageNet-S$_{50}$PASS, PASCAL VOC pascalvoc, and COCO-Stuff coco-stuff). We present UO-SAM, which, for the first time, outperforms the top-specialized baselines on multiple USS benchmarks.
• Figure 2: The architecture of our UO-SAM. We introduce a universal organizer of segment anything model (UO-SAM). It consists of the local region optimizer (LRO) and the global region optimizer (GRO) modules to improve the USS baseline performance.
• Figure 3: Qualitative comparisons of USS on the validation sets of ImageNet-S$_{50}$PASS, PASCAL VOC pascalvoc and COCO-Stuff coco-stuff.