Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors

TL;DR

This work tackles the challenge of repurposing diffusion priors for perception tasks that require different semantic granularities. It introduces Vermouth, a lightweight framework that freezes a pre-trained Stable Diffusion backbone, uses a U-head to fuse multi-granularity latent features across diffusion time steps $t$ and SD stages, and integrates discriminative priors via an Adapted-Expert guided by BLIP captions and CLIP embeddings. By evaluating on zero-shot sketch-based image retrieval, open-vocabulary semantic segmentation, and few-shot classification, Vermouth demonstrates strong, task-agnostic representations with a small trainable footprint, highlighting diffusion priors as versatile learners for perception. The approach offers a practical blueprint for leveraging diffusion models in discriminative tasks without heavy decoders and points to broader opportunities for diffusion-based perception research.

Abstract

The remarkable prowess of diffusion models in image generation has spurred efforts to extend their application beyond generative tasks. However, a persistent challenge exists in lacking a unified approach to apply diffusion models to visual perception tasks with diverse semantic granularity requirements. Our purpose is to establish a unified visual perception framework, capitalizing on the potential synergies between generative and discriminative models. In this paper, we propose Vermouth, a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. We emphasize that there is no necessity for incorporating a heavyweight or intricate decoder to transform diffusion models into potent representation learners. Extensive comparative evaluations against tailored discriminative models showcase the efficacy of our approach on zero-shot sketch-based image retrieval (ZS-SBIR), few-shot classification, and open-vocabulary semantic segmentation tasks. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.

Figures (12)

• Figure 1: We transfer priors of the SD model in a unified framework for different visual perception tasks.
• Figure 2: An overview of our framework. We employ the pre-trained BLIP model to acquire an accurate description and utilize the SD model to generate representations guided by text embedding. The introduction of the U-head is intended to fuse the two representations from the SD model and the adapted expert, aiming to enhance compatibility in discriminative tasks with different semantic granularity requirements.
• Figure 3: Feature visualization of different categories of sketches and images in different time steps.
• Figure 4: Time step
• Figure 5: Stage in U-net