Segment Anything in High Quality

TL;DR

The paper addresses the limitation of SAM's coarse masks by introducing HQ-SAM, a lightweight extension that yields high-quality segmentation without sacrificing zero-shot prompting. It achieves this via a learnable HQ-Output Token and global-local feature fusion, trained on a compact HQSeg-44K dataset while freezing SAM’s weights. HQ-SAM demonstrates substantial improvements across 10 diverse datasets, including zero-shot and video benchmarks, with minimal overhead and fast training (4 hours on 8 GPUs). The approach preserves SAM's generalization, enhances boundary accuracy, and offers practical value for open-world segmentation tasks and downstream editing applications.

Abstract

The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out of them are evaluated in a zero-shot transfer protocol. Our code and pretrained models are at https://github.com/SysCV/SAM-HQ.

Figures (14)

• Figure 1: The predicted masks of SAM vs. our HQ-SAM, given the same red box or several points on the object as input prompts. HQ-SAM produces significantly more detailed results with very accurate boundaries. In the rightmost column, SAM misinterprets the thin structure of the kite lines, and produces a large portion of errors with broken holes for the input box prompt.
• Figure 2: Performance vs. speed vs. model size for an array of SAM variants sammobile_sam.
• Figure 3: HQ-SAM introduces HQ-Output Token and Global-local Feature Fusion to SAM for high-quality mask prediction. To keep the zero-shot capability of SAM, the lightweight HQ-Output Token reuses SAM's mask decoder, and generates new MLP layers for performing point-wise product with fused HQ-Features. During training, only a few learnable parameters in HQ-SAM are trainable while we fix the model parameters of the pre-trained SAM. The prompt encoder is omitted here for clarity. Error correction is simply used as a direct element-wise sum between the predicted logits of the SAM's Output Token and the HQ-Output Token during inference.
• Figure 4: Recall rate comparison between COIFT liew2021deep and HRSOD zeng2019towards under the zero-shot protocol, using BIoU thresholds ranging from loose to strict. The performance gap between SAM and our HQ-SAM increases significantly when we vary from a loose BIoU threshold of 0.5 to a very strict threshold of 0.9, showing the advantage of HQ-SAM in predicting very accurate segmentation masks.
• Figure 5: Interactive segmentation results comparison using a varying number of input points on the COIFT liew2021deep (zero-shot) and DIS qin2022 val set. HQ-SAM consistently outperforms SAM with various point numbers, and the relative improvement is more obvious with less prompt ambiguity.