Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

Paul Couairon, Mustafa Shukor, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome

TL;DR

DiffCut introduces a zero-shot semantic segmentation method that leverages the final self-attention features of a diffusion model's UNet encoder to build a patch-level affinity. A recursive Normalized Cut on these diffusion features yields dense segmentation maps at a latent resolution, which are upsampled to high resolution through a concept-embedding assignment refined by PAMR. Across six benchmarks, DiffCut achieves state-of-the-art unsupervised performance, demonstrates strong semantic coherence of diffusion features, and offers an open-vocabulary extension via CLIP. The approach reduces reliance on labeled data and prompts, scales with content, and positions diffusion-based encoders as effective foundation models for downstream vision tasks, while noting domain-specific limitations and avenues for future enhancement.

Abstract

Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. While prior works have addressed unsupervised image segmentation, they significantly lag behind supervised models. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that the utilization of these diffusion features in a graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage a recursive Normalized Cut algorithm that softly regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks. Project page at https://diffcut-segmentation.github.io

DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

TL;DR

DiffCut introduces a zero-shot semantic segmentation method that leverages the final self-attention features of a diffusion model's UNet encoder to build a patch-level affinity. A recursive Normalized Cut on these diffusion features yields dense segmentation maps at a latent resolution, which are upsampled to high resolution through a concept-embedding assignment refined by PAMR. Across six benchmarks, DiffCut achieves state-of-the-art unsupervised performance, demonstrates strong semantic coherence of diffusion features, and offers an open-vocabulary extension via CLIP. The approach reduces reliance on labeled data and prompts, scales with content, and positions diffusion-based encoders as effective foundation models for downstream vision tasks, while noting domain-specific limitations and avenues for future enhancement.

Abstract

Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. While prior works have addressed unsupervised image segmentation, they significantly lag behind supervised models. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that the utilization of these diffusion features in a graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage a recursive Normalized Cut algorithm that softly regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks. Project page at https://diffcut-segmentation.github.io
Paper Structure (50 sections, 3 equations, 12 figures, 13 tables, 1 algorithm)

This paper contains 50 sections, 3 equations, 12 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic segmentation.
  4. Segmentation with Text Supervision.
  5. Graph-based Object Detection.
  6. Segmentation with Diffusion Models.
  7. DiffCut
  8. Diffusion Models.
  9. Features Extraction
  10. Attention Limitations.
  11. UNet Encoder Effectiveness.
  12. Computational Efficiency.
  13. Recursive Feature Clustering
  14. Graph Affinity.
  15. Recursive Partitioning.
  16. ...and 35 more sections

Figures (12)

  • Figure 2: Overview of DiffCut.1) DiffCut takes an image as input and extracts the features of the last self-attention block of a diffusion UNet encoder. 2) These features are used to construct an affinity matrix that serves in a recursive normalized cut algorithm, which outputs a segmentation map at the latent spatial resolution. 3) A high-resolution segmentation map is produced via a concept assignment mechanism on the features upsampled at the original image size.
  • Figure 3: ROC curves revealing the semantic coherence of vision encoders.
  • Figure 4: Qualitative results on the semantic coherence of various vision encoders. We select a patch (red marker) associated to the dog in Ref. image. Top row shows the cosine similarity heatmap between the selected patch and all patches produced by vision encoders for Ref. image. Bottom row shows the heatmap between the selected patch in Ref. image and all patches produced by vision encoders for Target. image.
  • Figure 5: Sensitivity of DiffCut. As $\alpha$ increases, DiffCut shows competitive results for a broad range of $\tau$ values.
  • Figure 6: Effect of $\tau$. As $\tau$ corresponds to the maximum Ncut value, a larger threshold loosens the constraint on the partitioning algorithm and allows it to perform more recursive steps to uncover finer objects. It can be interpreted as the level of granularity of detected objects.
  • ...and 7 more figures