Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, Chunhua Shen

TL;DR

This work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation, establishing a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior.

Abstract

The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.

Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

TL;DR

This work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation, establishing a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior.

Abstract

The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.
Paper Structure (22 sections, 13 equations, 8 figures, 7 tables)

This paper contains 22 sections, 13 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Method
  5. Interaction between query and support images
  6. Injection of support mask information
  7. Supervision from query mask
  8. Exploration of generation process
  9. 1-shot to N-shot
  10. Experiment
  11. In-context setting
  12. Strict few-shot setting
  13. Visualization
  14. Conclusion
  15. Appendix
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Overview of the DiffewS framework. (a)(b) display that query image $\mathbf{I}_q$, query mask $\mathbf{M}_q$, support image $\mathbf{I}_s$ and support mask $\mathbf{M}_s$ are all encoded by VAE into latent variables $\mathbf{z}_q$, $\mathbf{z}_{mq}$, $\mathbf{z}_s$, $\mathbf{z}_{ms}$, respectively, where $\mathbf{z}_q$ and $\mathbf{z}_{mq}$ are concatenated to input into UNet. (c) demonstrates the DiffewS fintuning protocol (d) elucidates the detailed implementation of FSA, acquiring information from support images by concatenating the query and key features.
  • Figure 2: Exploring the Interaction and Injection Methods
  • Figure 3: Illustrations and comparisons of different forms of supervision from query mask.
  • Figure 4: Illustrations and comparisons of different mask generation processes.
  • Figure 5: Qualitative results of one-shot semantic segmentation on LVIS-92$^i$. The blue color denotes the support mask while the red represents the query mask.
  • ...and 3 more figures