Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning

Duojun Huang, Xinyu Xiong, Jie Ma, Jichang Li, Zequn Jie, Lin Ma, Guanbin Li

TL;DR

This work introduces AlignSAM, a framework that adapts the Segment Anything Model to open contexts without altering SAM’s parameters. It employs a reinforcement-learning prompting agent to select informative patch-level prompts, progressively refining segmentation, while a semantic recalibration module provides reliable labels and supports both explicit and implicit semantics through explicit and implicit branches. The approach combines patch-level action spaces, PPO-based training, and cross-modal cues (via CLIP-inspired attention) to achieve robust, task-aware prompting. Empirical results across blur, shadow, glass, saliency, and Pascal-VOC tasks show AlignSAM outperforming state-of-the-art efficient-tuning and prompting methods, with ablations confirming the critical roles of RL, SRM, and multi-step prompting in driving gains.

Abstract

Powered by massive curated training data, Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts. However, the vanilla SAM is class agnostic and heavily relies on user-provided prompts to segment objects of interest. Adapting this method to diverse tasks is crucial for accurate target identification and to avoid suboptimal segmentation results. In this paper, we propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context through reinforcement learning. Anchored by an agent, AlignSAM enables the generality of the SAM model across diverse downstream tasks while keeping its parameters frozen. Specifically, AlignSAM initiates a prompting agent to iteratively refine segmentation predictions by interacting with the foundational model. It integrates a reinforcement learning policy network to provide informative prompts to the foundational models. Additionally, a semantic recalibration module is introduced to provide fine-grained labels of prompts, enhancing the model's proficiency in handling tasks encompassing explicit and implicit semantics. Experiments conducted on various challenging segmentation tasks among existing foundation models demonstrate the superiority of the proposed AlignSAM over state-of-the-art approaches. Project page: \url{https://github.com/Duojun-Huang/AlignSAM-CVPR2024}.

AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning

TL;DR

This work introduces AlignSAM, a framework that adapts the Segment Anything Model to open contexts without altering SAM’s parameters. It employs a reinforcement-learning prompting agent to select informative patch-level prompts, progressively refining segmentation, while a semantic recalibration module provides reliable labels and supports both explicit and implicit semantics through explicit and implicit branches. The approach combines patch-level action spaces, PPO-based training, and cross-modal cues (via CLIP-inspired attention) to achieve robust, task-aware prompting. Empirical results across blur, shadow, glass, saliency, and Pascal-VOC tasks show AlignSAM outperforming state-of-the-art efficient-tuning and prompting methods, with ablations confirming the critical roles of RL, SRM, and multi-step prompting in driving gains.

Abstract

Powered by massive curated training data, Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts. However, the vanilla SAM is class agnostic and heavily relies on user-provided prompts to segment objects of interest. Adapting this method to diverse tasks is crucial for accurate target identification and to avoid suboptimal segmentation results. In this paper, we propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context through reinforcement learning. Anchored by an agent, AlignSAM enables the generality of the SAM model across diverse downstream tasks while keeping its parameters frozen. Specifically, AlignSAM initiates a prompting agent to iteratively refine segmentation predictions by interacting with the foundational model. It integrates a reinforcement learning policy network to provide informative prompts to the foundational models. Additionally, a semantic recalibration module is introduced to provide fine-grained labels of prompts, enhancing the model's proficiency in handling tasks encompassing explicit and implicit semantics. Experiments conducted on various challenging segmentation tasks among existing foundation models demonstrate the superiority of the proposed AlignSAM over state-of-the-art approaches. Project page: \url{https://github.com/Duojun-Huang/AlignSAM-CVPR2024}.
Paper Structure (37 sections, 14 equations, 7 figures, 10 tables, 1 algorithm)

This paper contains 37 sections, 14 equations, 7 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image segmentation.
  4. Vision foundation models.
  5. Parameter-Efficient Fine-Tuning.
  6. Reinforcement learning for computer vision.
  7. Methodology
  8. Preliminary
  9. Revisit of Segment Anything Model.
  10. Target-aware Reinforcement Learning
  11. Action space.
  12. State space.
  13. Reward function.
  14. Training RL model.
  15. Semantic Recalibration Module
  16. ...and 22 more sections

Figures (7)

  • Figure 1: Conceptual comparisons of our method and previous approaches. Frozen and learnable parameters are highlighted in blue and orange, respectively. (a) The proposed method. (b) Text-guided methods arXiv23_RefSAMarXiv23_SAMCLIP. (c) PEFT methods ICCVW23_SAM-AdapterarXiv23_CustomSAM. (d) In-context learning methods arXiv23_PerSAMarXiv23_Matcher. Observed that the proposed agent-based auto-prompting effectively grasps vision and linguistic cues, unleashing the potential of the foundation segmentation model in various contexts, such as saliency detection, shadow detection, blur detection, and glass detection.
  • Figure 2: Segmentation results for CLIP-Surgery arXiv2023_sclip using different prompts. (a) and (b) illustrate the prompts of "dog" and "salient object", respectively. Observed that vision-language models excel in processing explicit semantics but often struggle with implicit semantics.
  • Figure 3: An overview of the proposed AlignSAM, which consists of a Segment Anything Model (SAM), a vision-language model (CLIP-Surgery arXiv2023_sclip), a reinforcement learning agent, and a semantic recalibration module. The frozen SAM receives the point prompts generated by the agent and semantic recalibration module, dealing with various downstream scenarios without relying on manual prompting.
  • Figure 4: Qualitative comparisons between our and other methods. (a) Blur detection. (b) Shadow detection. (c) Glass detection. (d) Semantic segmentation.
  • Figure 5: Several examples to illustrate the iterative point selection and the corresponding segmentation results. The sequence progresses from left to right, showing a gradual increase in the number of point prompts.
  • ...and 2 more figures