Generalization Boosted Adapter for Open-Vocabulary Segmentation

TL;DR

This work tackles open-vocabulary segmentation with CLIP-based backbones, identifying overfitting and insufficient pixel-level nuance when fine-tuning on limited data. It introduces Generalization Boosted Adapter (GBA), a dual-adapter framework consisting of Style Diversification Adapter (SDA) and Correlation Constraint Adapter (CCA) that operate at different depths to diversify features and enforce semantic alignment. SDA performs frequency-domain amplitude manipulation to enrich style while preserving content, while CCA uses cross-attention and frequency-domain normalization to emphasize high-frequency semantic cues and suppress false correlations. Together, GBA delivers state-of-the-art results on open-vocabulary panoptic and semantic segmentation benchmarks with minimal computational overhead and broad plug-and-play compatibility, including benefits on transformer-based CLIP variants. The approach demonstrates robust generalization across diverse datasets and offers a practical path for enhancing cross-modal segmentation tasks in real-world settings.

Abstract

Vision-language models (VLMs) have demonstrated remarkable open-vocabulary object recognition capabilities, motivating their adaptation for dense prediction tasks like segmentation. However, directly applying VLMs to such tasks remains challenging due to their lack of pixel-level granularity and the limited data available for fine-tuning, leading to overfitting and poor generalization. To address these limitations, we propose Generalization Boosted Adapter (GBA), a novel adapter strategy that enhances the generalization and robustness of VLMs for open-vocabulary segmentation. GBA comprises two core components: (1) a Style Diversification Adapter (SDA) that decouples features into amplitude and phase components, operating solely on the amplitude to enrich the feature space representation while preserving semantic consistency; and (2) a Correlation Constraint Adapter (CCA) that employs cross-attention to establish tighter semantic associations between text categories and target regions, suppressing irrelevant low-frequency ``noise'' information and avoiding erroneous associations. Through the synergistic effect of the shallow SDA and the deep CCA, GBA effectively alleviates overfitting issues and enhances the semantic relevance of feature representations. As a simple, efficient, and plug-and-play component, GBA can be flexibly integrated into various CLIP-based methods, demonstrating broad applicability and achieving state-of-the-art performance on multiple open-vocabulary segmentation benchmarks.

Figures (8)

• Figure 1: We use FC-CLIP as the baseline to investigate the performance of different adapter methods in dense prediction tasks. As shown in (a), the model with plain adapter incorrectly identifies the object as a 'sofa' and further misclassifies the 'wall' as a "rug" due to false associations. In contrast, our proposed GBA method accurately recognizes the 'shelf' and 'wall' in the image, as illustrated in (b). This improvement can be attributed to the GBA's ability to enhance the model's generalization capability and suppress false associations for dense prediction tasks.
• Figure 2: Overview of the GBA framework. Single-stage open-vocabulary segmentation methods akin to FC-CLIP comprise three key components: a mask generator, an in-vocabulary classifier, and an out-of-vocabulary classifier. All these components are constructed based on features extracted from a frozen CNN-based CLIP backbone that employs the proposed two learning feature augmentation adapters, namely, the Style Diversification Adapter (SDA) and the Correlation Constraint Adapter (CCA).
• Figure 3: The detail of plain adapter. Adapters exhibiting diverse generalization capacities can be realized through the utilization of varied feature enhenced strategies.
• Figure 4: The details of Style Diversification Adapter (SDA) (a) and Correlation Constraint Adapter (CCA) (b).
• Figure 5: Qualitative Visualization of Open-Vocabulary Panoptic Segmentation. To showcase the open-vocabulary recognition capability, we amalgamated class names from all datasets totaling approximately all classes, and conducted open-vocabulary inference directly.