Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

Boyu Han, Qianqian Xu, Shilong Bao, Zhiyong Yang, Ruochen Cui, Xilin Zhao, Qingming Huang

TL;DR

The Diffusion Contrastive Reconstruction (DCR) is introduced, which unifies the learning objective and shows that the DCR loss can jointly optimize D-Ability and P-Ability.

Abstract

The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

TL;DR

The Diffusion Contrastive Reconstruction (DCR) is introduced, which unifies the learning objective and shows that the DCR loss can jointly optimize D-Ability and P-Ability.

Abstract

The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.
Paper Structure (30 sections, 3 theorems, 103 equations, 8 figures, 10 tables, 1 algorithm)

This paper contains 30 sections, 3 theorems, 103 equations, 8 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Diffusion Models
  5. The Task of Visual Representation Learning
  6. Methodology
  7. A Naive Optimization Method
  8. Diffusion Contrastive Reconstruction
  9. Theoretical Statements
  10. Experiments
  11. Experimental Setups
  12. Overall Performance
  13. Multimodal Large Language Model Evaluation
  14. Ablation Studies
  15. Conclusion
  16. ...and 15 more sections

Key Result

Theorem 1

Fix a diffusion step $t$ and define $T(\mathbf{z})=\epsilon_{\theta}\bigl(x_t,\,h_{\omega}(\mathbf{z}),\,t\bigr)$. Under a mild assumption, the intra-class scatter and inter-class scatter in the feature space ($S_{\text{inner}}, S_{\text{inter}}$) can be bounded by those in the noise space ($S_{\tex where $m$, $\kappa$, and $\eta$ are positive constants depending on the Lipschitz continuity of $T(

Figures (8)

  • Figure 1: (a) Contrastive learning for D-Ability. (b) Reconstructive learning for P-Ability. (c) Our Diffusion Contrastive Reconstruction (DCR) for harmonizing D-Ability and P-Ability. (d) Performance overview of DCR and other methods on the OpenAI CLIP ViT-L@224 backbone.
  • Figure 2: Training results of linearly combining contrastive learning and reconstructive learning (Naive Method) on the OpenAI CLIP ViT-L@224 backbone. The contrastive loss $\mathcal{L}_{\text{con}}$ dominates the gradients, and most training steps exhibit gradient conflicts.
  • Figure 3: An overview of Diffusion Contrastive Reconstruction (DCR). An image is encoded by CLIP and projected into the diffusion condition space. Predicted noises from original, augmented, and negative samples form a contrastive triplet in the reconstruction image space. Training proceeds in two stages: projector alignment and encoder enhancement.
  • Figure 4: Qualitative results of P-Ability on the MMVP-VLM benchmark. The predictions from the original CLIP and our improved version are indicated by red and green arrows, respectively. The improved CLIP effectively addresses the original model's limitations in capturing fine-grained visual details.
  • Figure 5: Qualitative results of D-Ability on the MNIST benchmark by using the t-SNE method. The improved CLIP achieves better class separability.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Remark 1
  • Theorem 1
  • Theorem 2
  • Remark 2
  • Definition 1: Bi-Lipschitz Mapping david2016bi
  • Lemma 1: Equivalent Variance Identity zhang2012some
  • proof
  • proof
  • proof