Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation

TL;DR

This work tackles open-vocabulary semantic segmentation by addressing cross-modal feature gaps that cause false matches. It introduces RIM, a training-free framework that builds well-aligned intra-modal references from Stable Diffusion and Segment Anything Model and performs relation-aware matching in the all-purpose DINOv2 feature space. A ranking-distribution formulation leverages inter-class relationships and subcategory references to handle intra-class diversity. Experiments on VOC and COCO demonstrate large improvements over prior zero-shot and training-free methods, indicating a practical and scalable path for open vocabulary segmentation.

Abstract

Open-vocabulary semantic segmentation (OVS) aims to segment images of arbitrary categories specified by class labels or captions. However, most previous best-performing methods, whether pixel grouping methods or region recognition methods, suffer from false matches between image features and category labels. We attribute this to the natural gap between the textual features and visual features. In this work, we rethink how to mitigate false matches from the perspective of image-to-image matching and propose a novel relation-aware intra-modal matching (RIM) framework for OVS based on visual foundation models. RIM achieves robust region classification by firstly constructing diverse image-modal reference features and then matching them with region features based on relation-aware ranking distribution. The proposed RIM enjoys several merits. First, the intra-modal reference features are better aligned, circumventing potential ambiguities that may arise in cross-modal matching. Second, the ranking-based matching process harnesses the structure information implicit in the inter-class relationships, making it more robust than comparing individually. Extensive experiments on three benchmarks demonstrate that RIM outperforms previous state-of-the-art methods by large margins, obtaining a lead of more than 10% in mIoU on PASCAL VOC benchmark.

Figures (6)

• Figure 1: Motivation of our method. (a) False matches tend to occur in cross-modal features. We establish well-aligned image-modal reference features thus transit the text-to-image matching to image-to-image matching. (b) Indivdual matching tends to suffer from disturbances. We propose a novel relation-aware matching strategy for more robust region classification.
• Figure 2: Framework of our proposed Relation-aware Intra-modal Matching (RIM) Network. We first explore Stable Diffusion model and SAM to construct image-modal reference features, then we conduct relation-aware matching between region features and reference features based on ranking distribution. The matching is established in the all-purpose feature spaces of DINOv2.
• Figure 3: Qualitative results of our method.
• Figure 4: Effectiveness of SAM based region-level matching. The SAM could well capture visual concepts within images.
• Figure 5: Hyperparameter experiments on the number of subcategories and binarization threshold of cross-attention map.