Pro2SAM: Mask Prompt to SAM with Grid Points for Weakly Supervised Object Localization

TL;DR

Pro2SAM addresses weakly supervised object localization by combining the zero-shot, fine-grained segmentation capabilities of the Segment Anything Model (SAM) with a novel Global Token Transformer (GTFormer) that produces a coarse, localization-like mask from image-level labels. It then uses dense grid-point prompts to generate a mask gallery from SAM and applies pixel-level similarity to match the coarse prompt to a precise foreground mask, effectively resolving semantic ambiguity in SAM prompts. The GTFormer employs global tokens and global-aware transformer blocks to capture global–local dependencies, yielding a robust coarse localization that guides mask matching. Across CUB-200-2011 and ILSVRC, Pro2SAM achieves state-of-the-art localization performance, with strong fine-grained segmentation capabilities and scalable inference, demonstrating the practical impact of integrating foundation-model segmentation with targeted prompt learning for WSOL.

Abstract

Weakly Supervised Object Localization (WSOL), which aims to localize objects by only using image-level labels, has attracted much attention because of its low annotation cost in real applications. Current studies focus on the Class Activation Map (CAM) of CNN and the self-attention map of transformer to identify the region of objects. However, both CAM and self-attention maps can not learn pixel-level fine-grained information on the foreground objects, which hinders the further advance of WSOL. To address this problem, we initiatively leverage the capability of zero-shot generalization and fine-grained segmentation in Segment Anything Model (SAM) to boost the activation of integral object regions. Further, to alleviate the semantic ambiguity issue accrued in single point prompt-based SAM, we propose an innovative mask prompt to SAM (Pro2SAM) network with grid points for WSOL task. First, we devise a Global Token Transformer (GTFormer) to generate a coarse-grained foreground map as a flexible mask prompt, where the GTFormer jointly embeds patch tokens and novel global tokens to learn foreground semantics. Secondly, we deliver grid points as dense prompts into SAM to maximize the probability of foreground mask, which avoids the lack of objects caused by a single point/box prompt. Finally, we propose a pixel-level similarity metric to come true the mask matching from mask prompt to SAM, where the mask with the highest score is viewed as the final localization map. Experiments show that the proposed Pro2SAM achieves state-of-the-art performance on both CUB-200-2011 and ILSVRC, with 84.03\% and 66.85\% Top-1 Loc, respectively.

Figures (5)

• Figure 1: (a) Comparison of Ours, CAM-based (CAM ref-1 and BagCAM ref-42), and transformer-based methods (SAT ref-6 and UOLwRPS ref-46). The red and green boxes are GT bounding boxes and predicted bounding boxes, and box prompt, respectively. (b) Ambiguity of SAM, blue bounding box is box prompt and red star is point prompt.
• Figure 2: Overview of our proposed Pro2SAM framework, which comprises three stages: Stage 1) Mask Prompt Generation stage, we propose a Global Token Transformer (GTFormer) to predict coarse foreground maps via only image-level labels; Stage 2) SAM Assist Stage, we input the grid points of the image into SAM as prompts to generate all masks of input images; Stage 3) Mask Matching Stage, we evaluate the similarity scores between the coarse mask from GTFormer and fine-grained masks from SAM to select a perfect mask as a localization map.
• Figure 3: Overview of our proposed GTFormer, which inherits the architecture of transformer blocks for ViT ref-7ref-17 and adds novel global tokens and global-aware transformer blocks. Furthermore, we propose a global embedding layer to learn global tokens.
• Figure 4: Visualization comparison. The ground-truth bounding boxes are in red, and the predicted bounding boxes are in green.
• Figure 5: Accurate localization comparison of Pr2SAM and other SOTA methods with diverse IOU thresholds.