AoP-SAM: Automation of Prompts for Efficient Segmentation

TL;DR

AoP-SAM tackles the inefficiency of manual prompting in SAM by introducing a lightweight Prompt Predictor that uses SAM's image embeddings to generate a Prompt Confidence Map and a test-time Adaptive Sampling and Filtering (ASF) module to select essential prompts in a coarse-to-fine manner. Trained on the SA-1B dataset and evaluated across SA-1B, COCO, and LVIS with multiple SAM backbones, AoP-SAM achieves higher segmentation accuracy (mIoU) while maintaining competitive latency and memory usage compared to grid-based and detector-based baselines. The approach eliminates the need for human intervention in prompting, enabling faster automated segmentation in real-world, resource-constrained settings, including edge devices, without finetuning SAM. Overall, AoP-SAM advances practical zero-shot segmentation by coupling a tightly integrated prompt predictor with test-time adaptation to minimize redundant prompts and computations.

Abstract

The Segment Anything Model (SAM) is a powerful foundation model for image segmentation, showing robust zero-shot generalization through prompt engineering. However, relying on manual prompts is impractical for real-world applications, particularly in scenarios where rapid prompt provision and resource efficiency are crucial. In this paper, we propose the Automation of Prompts for SAM (AoP-SAM), a novel approach that learns to generate essential prompts in optimal locations automatically. AoP-SAM enhances SAM's efficiency and usability by eliminating manual input, making it better suited for real-world tasks. Our approach employs a lightweight yet efficient Prompt Predictor model that detects key entities across images and identifies the optimal regions for placing prompt candidates. This method leverages SAM's image embeddings, preserving its zero-shot generalization capabilities without requiring fine-tuning. Additionally, we introduce a test-time instance-level Adaptive Sampling and Filtering mechanism that generates prompts in a coarse-to-fine manner. This notably enhances both prompt and mask generation efficiency by reducing computational overhead and minimizing redundant mask refinements. Evaluations of three datasets demonstrate that AoP-SAM substantially improves both prompt generation efficiency and mask generation accuracy, making SAM more effective for automated segmentation tasks.

Figures (2)

• Figure 1: In SAM, automating prompt provision eliminates the need for manual input, significantly improving the efficiency of mask segmentation. However, current approaches, such as grid-based prompts in vanilla SAM (AMG_S for sparse, AMG_D for dense) or extra detection models (OAS: Box or Central Point), often introduce excessive mask refinements or computational overhead, leading to increased latency and reduced efficiency. In contrast, our proposed AoP-SAM efficiently generates essential prompts for accurate mask generation within SAM, entirely without human intervention. In the illustrations above, different colors represent various segmentation results, with orange labels (stars or boxes) indicating valid prompts, green labels marking invalid prompts, and black stars in our results representing the filtered prompts, processed in a coarse-to-fine manner by the test-time instance-wise Adaptive Sampling and Filtering (ASF) mechanism.
• Figure 2: The architecture of our proposed AoP-SAM consists of two key components: the prompt predictor and the Adaptive Sampler and Filter (ASF) Module. The prompt predictor operates by taking the image input and the computed image embedding from SAM's image encoder as inputs. Prompt predictor then generates a Prompt Confidence Map (PCM) that highlights potential regions for prompt candidates. During test-time, these candidates are adaptively sampled and filtered by ASF, predicting prompts that might lead to redundant masks based on the generated mask references. This process eliminates unnecessary prompts, ensuring that only the essential ones are used to generate the final mask results.