Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupled Classifier-Free Guidance for Counterfactual Diffusion Models

Tian Xia, Fabio De Sousa Ribeiro, Rajat R Rasal, Avinash Kori, Raghav Mehta, Ben Glocker

TL;DR

This paper tackles the problem that classifier-free guidance with a single global weight can cause unintended attribute amplification during counterfactual diffusion. It introduces Decoupled Classifier-Free Guidance (DCFG), a model-agnostic, inference-time technique that partitions semantic attributes into groups and applies distinct guidance weights per group via an attribute-split conditioning embedding. The approach reduces spurious changes outside the causal pathway while preserving the targeted intervention’s effect, demonstrated on CelebA-HQ and medical-imaging datasets (EMBED, MIMIC-CXR) with metrics for effectiveness and reversibility. The results show improved fidelity and reversibility of counterfactuals as guidance is decoupled, with flexible configurations from two-group to per-attribute control. The work suggests broad applicability of DCFG to other conditional generation tasks and timesteps scheduling strategies for further fidelity gains.

Abstract

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose Decoupled Classifier-Free Guidance (DCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. DCFG is implemented via a simple attribute-split embedding strategy that disentangles semantic inputs, enabling selective guidance on user-defined attribute groups.

Decoupled Classifier-Free Guidance for Counterfactual Diffusion Models

TL;DR

This paper tackles the problem that classifier-free guidance with a single global weight can cause unintended attribute amplification during counterfactual diffusion. It introduces Decoupled Classifier-Free Guidance (DCFG), a model-agnostic, inference-time technique that partitions semantic attributes into groups and applies distinct guidance weights per group via an attribute-split conditioning embedding. The approach reduces spurious changes outside the causal pathway while preserving the targeted intervention’s effect, demonstrated on CelebA-HQ and medical-imaging datasets (EMBED, MIMIC-CXR) with metrics for effectiveness and reversibility. The results show improved fidelity and reversibility of counterfactuals as guidance is decoupled, with flexible configurations from two-group to per-attribute control. The work suggests broad applicability of DCFG to other conditional generation tasks and timesteps scheduling strategies for further fidelity gains.

Abstract

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose Decoupled Classifier-Free Guidance (DCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. DCFG is implemented via a simple attribute-split embedding strategy that disentangles semantic inputs, enabling selective guidance on user-defined attribute groups.

Paper Structure

This paper contains 45 sections, 1 theorem, 27 equations, 28 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Structural Causal Models.
  4. Counterfactual inference.
  5. Diffusion Models for Counterfactual Inference
  6. Classifier-Free Guidance
  7. Decoupled Classifier-Free Guidance
  8. Attribute-Split Conditioning Embedding
  9. Formulation: Group-wise DCFG
  10. DCFG for Counterfactual Generation
  11. Experiments
  12. Case Study 1: CelebA
  13. Case Study 2: Mammography
  14. Case Study 3: Chest Radiography
  15. Conclusion
  16. ...and 30 more sections

Key Result

Proposition 1

Under the assumption that the groups $\mathbf{pa}^{(1)}, \dots, \mathbf{pa}^{(M)}$ are conditionally independent given the latent variable $\mathbf{x}_t$, for any time $t$, that is: $p(\mathbf{pa} \mid \mathbf{x}_t) = \prod_{m=1}^M p(\mathbf{pa}^{(m)} \mid \mathbf{x}_t)$, we obtain the following fac where $\omega_m \geq 0$ controls the guidance strength for each group $m$.

Figures (28)

  • Figure 1: Comparison of $\Delta$ metrics under different interventions in CelebA-HQ. Left: Intervention on Smiling. Right: Intervention on Young. Both use baseline $\omega{=}1.0$. Under global CFG, increasing $\omega$ boosts the intended attribute but amplifies non-target ones. DCFG achieves similar improvements on the target attribute while mitigating amplification. See section \ref{['app:extra_numeric_celeba']} for full quantitative results.
  • Figure 2: Counterfactual generations in CelebA-HQ ($64\times 64$). Each row compares global CFG (left) and DCFG (right) across guidance weights. Top: global CFG causes amplification of Smiling under $\texttt{do(Male)}$; Middle: $\texttt{do(Young)}$ suppresses Male (i.e. amplifies $\texttt{Male}{=}no$); Bottom: $\texttt{do(Smiling)}$ makes the subject appear older, adds glasses, and alters identity. DCFG mitigates these unintended changes and preserves invariant attributes. See section \ref{['app:extra_visual_celeba']} for more visual results.
  • Figure 3: Reversibility analysis in CelebA-HQ $(64\times 64)$. Left: Quantitative evaluation of how well the original image is recovered after generating a counterfactual and mapping it back to the original condition under $\texttt{do(Smiling)}$. Right: A qualitative example showing a counterfactual generated under $\texttt{do(Male)}$ and its reconstruction after reversing the intervention with CFG and our DCFG.
  • Figure 4: Qualitative results for $\texttt{do(Smiling, Male, Young)}$. We compare two-group DCFG ($\omega_{\mathrm{aff}}=2.5, \omega_{\mathrm{inv}}=1.0$) with attribute-wise DCFG, where $\omega_{s}$, $\omega_{m}$, and $\omega_{y}$ control guidance for Smiling, Male, and Young. Symmetric weights ($\omega_{s}=\omega_{m}=\omega_{y}=2.5$) reproduce two-group results, while asymmetric weights highlight DCFG’s flexibility. See \ref{['sec:triple_intervention']} for more results.
  • Figure 5: Evaluation of counterfactual generation on EMBED ($192\times192$). Left: $\Delta$ metrics showing the effect of do(circle). DCFG improves target intervention effectiveness while suppressing spurious shifts in non-intervened attributes. Right: A visual example showing the input image, the counterfactual under do(density), the reversed image, and their difference maps (CF/Rev. - input). See sec. \ref{['app:extra_numeric_embed']} for full quantitative results and sec. \ref{['app:extra_visual_embed']} for more visual results.
  • ...and 23 more figures

Theorems & Definitions (1)

  • Proposition 1: Proxy Posterior for DCFG