Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models

Barbara Toniella Corradini, Mustafa Shukor, Paul Couairon, Guillaume Couairon, Franco Scarselli, Matthieu Cord

TL;DR

This work tackles open-vocabulary image segmentation without any training data or annotated masks by exploiting internal representations of diffusion models and a captioner. The FreeSeg-Diff pipeline clusters multiscale diffusion features to create class-agnostic masks, then maps these masks to textual labels via CLIP using candidate classes derived from BLIP captions, followed by a refinement step with CRF. It demonstrates that a training-free approach using diffusion-model features can outperform several training-based methods on VOC and COCO, and remains competitive with recent weakly-supervised and diffusion-based approaches, while highlighting the continued gap to fully supervised SOTA. The study also provides extensive ablations, showing diffusion backbones offer superior localization, the importance of caption-guided candidate pruning, and the benefits of refinement modules, suggesting a promising path for leveraging foundation models in dense prediction without large-scale training.

Abstract

Foundation models have exhibited unprecedented capabilities in tackling many domains and tasks. Models such as CLIP are currently widely used to bridge cross-modal representations, and text-to-image diffusion models are arguably the leading models in terms of realistic image generation. Image generative models are trained on massive datasets that provide them with powerful internal spatial representations. In this work, we explore the potential benefits of such representations, beyond image generation, in particular, for dense visual prediction tasks. We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets, with pixel-level annotations. To avoid the annotation cost or training large diffusion models, we constraint our setup to be zero-shot and training-free. In a nutshell, our pipeline leverages different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. The pipeline is as follows: the image is passed to both a captioner model (i.e. BLIP) and a diffusion model (i.e., Stable Diffusion Model) to generate a text description and visual representation, respectively. The features are clustered and binarized to obtain class agnostic masks for each object. These masks are then mapped to a textual class, using the CLIP model to support open-vocabulary. Finally, we add a refinement step that allows to obtain a more precise segmentation mask. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets. In addition, we show very competitive results compared to the recent weakly-supervised segmentation approaches. We provide comprehensive experiments showing the superiority of diffusion model features compared to other pretrained models. Project page: https://bcorrad.github.io/freesegdiff/

FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models

TL;DR

This work tackles open-vocabulary image segmentation without any training data or annotated masks by exploiting internal representations of diffusion models and a captioner. The FreeSeg-Diff pipeline clusters multiscale diffusion features to create class-agnostic masks, then maps these masks to textual labels via CLIP using candidate classes derived from BLIP captions, followed by a refinement step with CRF. It demonstrates that a training-free approach using diffusion-model features can outperform several training-based methods on VOC and COCO, and remains competitive with recent weakly-supervised and diffusion-based approaches, while highlighting the continued gap to fully supervised SOTA. The study also provides extensive ablations, showing diffusion backbones offer superior localization, the importance of caption-guided candidate pruning, and the benefits of refinement modules, suggesting a promising path for leveraging foundation models in dense prediction without large-scale training.

Abstract

Foundation models have exhibited unprecedented capabilities in tackling many domains and tasks. Models such as CLIP are currently widely used to bridge cross-modal representations, and text-to-image diffusion models are arguably the leading models in terms of realistic image generation. Image generative models are trained on massive datasets that provide them with powerful internal spatial representations. In this work, we explore the potential benefits of such representations, beyond image generation, in particular, for dense visual prediction tasks. We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets, with pixel-level annotations. To avoid the annotation cost or training large diffusion models, we constraint our setup to be zero-shot and training-free. In a nutshell, our pipeline leverages different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. The pipeline is as follows: the image is passed to both a captioner model (i.e. BLIP) and a diffusion model (i.e., Stable Diffusion Model) to generate a text description and visual representation, respectively. The features are clustered and binarized to obtain class agnostic masks for each object. These masks are then mapped to a textual class, using the CLIP model to support open-vocabulary. Finally, we add a refinement step that allows to obtain a more precise segmentation mask. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets. In addition, we show very competitive results compared to the recent weakly-supervised segmentation approaches. We provide comprehensive experiments showing the superiority of diffusion model features compared to other pretrained models. Project page: https://bcorrad.github.io/freesegdiff/
Paper Structure (29 sections, 4 figures, 6 tables)

This paper contains 29 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Diffusion Models
  4. Open-Vocabulary Zero-Shot Segmentation
  5. Leveraging diffusion models for visual tasks
  6. FreeSeg-Diff
  7. Candidate classes assessment.
  8. Class-agnostic masks retrieval.
  9. Masks classification and refinement.
  10. Experiments
  11. Implementation Details
  12. Comparison with State-of-The-Art Methods
  13. Mask supervised approaches.
  14. Text weakly-supervised approaches.
  15. Training-free text-based approaches.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Our detailed pipeline for zero-shot semantic segmentation. Image features extracted from the DM blocks are clustered to create class-agnostic masks, which are then superimposed on the original image and processed by CLIP to link each mask to a specific text (i.e., class). Textual classes are derived from entities extracted from a BLIP caption. After matching masks to classes, more accurate masks are obtained by a mask refinement module. All models remain frozen.
  • Figure 2: Qualitative segmentation results of FreeSeg-Diff.Closed-vocabulary segmentation (\ref{['fig:voc_multi_qual_results']} and \ref{['fig:coco_qual_results']}). From top to bottom: original image, clustering of DM features, and final segmentation. Our pipeline filters out redundant clusters while retaining key objects and refines coarse masks to yield sharp segmentation maps. Open-vocabulary segmentation (\ref{['fig:openvoc-qual']})
  • Figure 2: Impact of each stage of FreeSeg-Diff. We evaluate on Pascal VOC the baseline (SDM and CLIP, without caption support), then we add the captioning module (BLIP), the refinement module and the stable diffusion attention maps (Attn).
  • Figure 3: Clustering features from different pre-trained backbones. Stable diffusion model exhibits more semantic-aware internal features compared to those of CLIP-ViT, DINOv2 or VIT.