Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation

TL;DR

Open-vocabulary segmentation faces the challenge of localizing arbitrary text-described concepts without costly training. FreeDA introduces a training-free pipeline that builds offline diffusion-augmented textual-visual references and leverages both local (region-level) and global (image-level) semantics at inference to assign labels. Key contributions are the diffusion-based offline prototype generation, a non-parametric retrieval and matching mechanism, and extensive state-of-the-art results on five benchmarks without any training. The approach offers a practical, explainable alternative to supervised or contrastive training for open-vocabulary segmentation with strong empirical impact.

Abstract

Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form. Previous works have trained over large amounts of image-caption pairs to enforce pixel-level multimodal alignments. However, captions provide global information about the semantics of a given image but lack direct localization of individual concepts. Further, training on large-scale datasets inevitably brings significant computational costs. In this paper, we propose FreeDA, a training-free diffusion-augmented method for open-vocabulary semantic segmentation, which leverages the ability of diffusion models to visually localize generated concepts and local-global similarities to match class-agnostic regions with semantic classes. Our approach involves an offline stage in which textual-visual reference embeddings are collected, starting from a large set of captions and leveraging visual and semantic contexts. At test time, these are queried to support the visual matching process, which is carried out by jointly considering class-agnostic regions and global semantic similarities. Extensive analyses demonstrate that FreeDA achieves state-of-the-art performance on five datasets, surpassing previous methods by more than 7.0 average points in terms of mIoU and without requiring any training.

Figures (10)

• Figure 1: Open-vocabulary segmentation with: (a) TCL cha2023learning, which performs end-to-end learning of region-text alignment; (b) our FreeDA, which leverages generated textual-visual embeddings with global-local similarities and does not require any training.
• Figure 2: Overview of the diffusion-augmented prototype generation phase of FreeDA. Visual prototypes are generated by pooling self-supervised visual features on weak localization masks extracted from Stable Diffusion.
• Figure 3: Overview of the inference process in FreeDA. Local (region-level) and global similarities are computed by employing, respectively, visual self-supervised and multimodal contrastive embedding spaces, and by comparing them with input texts and prototypes, built during the off-line stage.
• Figure 4: Qualitative results of FreeDA in comparison with TCL cha2023learning, with and without global similarities and superpixels.
• Figure 5: Retrieval results when using an approximate index (left) and varying the number of retrieved key-prototype pairs (right).