Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Tilted Seesaw: Revisiting Autoencoder Trade-off for Controllable Diffusion

Pu Cao, Yiyang Ma, Feng Zhou, Xuedan Yin, Qing Song, Lu Yang

TL;DR

The paper addresses how a generation-centric bias in autoencoder evaluation skews latent-diffusion benchmarks and jeopardizes controllability. It formalizes autoencoder-induced condition drift $Δ_{AE}(x)$ and proves an alignment bound $L_{align}(θ) = E_x[Δ_{AE}(x)]$ that bounds controllability under latent-optimum diffusion, highlighting an irreducible limit. Through theory and extensive ImageNet AE experiments, it shows gFID weakly predicts drift while instance-level reconstruction metrics (e.g., PSNR) align strongly with condition preservation; ControlNet experiments and latent-condition probes corroborate that controllability tracks condition preservation rather than gFID. The authors propose reporting generative quality alongside at least one instance-level reconstruction metric and explicit condition-consistency measures to improve benchmarking and model selection for scalable controllable diffusion.

Abstract

In latent diffusion models, the autoencoder (AE) is typically expected to balance two capabilities: faithful reconstruction and a generation-friendly latent space (e.g., low gFID). In recent ImageNet-scale AE studies, we observe a systematic bias toward generative metrics in handling this trade-off: reconstruction metrics are increasingly under-reported, and ablation-based AE selection often favors the best-gFID configuration even when reconstruction fidelity degrades. We theoretically analyze why this gFID-dominant preference can appear unproblematic for ImageNet generation, yet becomes risky when scaling to controllable diffusion: AEs can induce condition drift, which limits achievable condition alignment. Meanwhile, we find that reconstruction fidelity, especially instance-level measures, better indicates controllability. We empirically validate the impact of tilted autoencoder evaluation on controllability by studying several recent ImageNet AEs. Using a multi-dimensional condition-drift evaluation protocol reflecting controllable generation tasks, we find that gFID is only weakly predictive of condition preservation, whereas reconstruction-oriented metrics are substantially more aligned. ControlNet experiments further confirm that controllability tracks condition preservation rather than gFID. Overall, our results expose a gap between ImageNet-centric AE evaluation and the requirements of scalable controllable diffusion, offering practical guidance for more reliable benchmarking and model selection.

A Tilted Seesaw: Revisiting Autoencoder Trade-off for Controllable Diffusion

TL;DR

The paper addresses how a generation-centric bias in autoencoder evaluation skews latent-diffusion benchmarks and jeopardizes controllability. It formalizes autoencoder-induced condition drift and proves an alignment bound that bounds controllability under latent-optimum diffusion, highlighting an irreducible limit. Through theory and extensive ImageNet AE experiments, it shows gFID weakly predicts drift while instance-level reconstruction metrics (e.g., PSNR) align strongly with condition preservation; ControlNet experiments and latent-condition probes corroborate that controllability tracks condition preservation rather than gFID. The authors propose reporting generative quality alongside at least one instance-level reconstruction metric and explicit condition-consistency measures to improve benchmarking and model selection for scalable controllable diffusion.

Abstract

In latent diffusion models, the autoencoder (AE) is typically expected to balance two capabilities: faithful reconstruction and a generation-friendly latent space (e.g., low gFID). In recent ImageNet-scale AE studies, we observe a systematic bias toward generative metrics in handling this trade-off: reconstruction metrics are increasingly under-reported, and ablation-based AE selection often favors the best-gFID configuration even when reconstruction fidelity degrades. We theoretically analyze why this gFID-dominant preference can appear unproblematic for ImageNet generation, yet becomes risky when scaling to controllable diffusion: AEs can induce condition drift, which limits achievable condition alignment. Meanwhile, we find that reconstruction fidelity, especially instance-level measures, better indicates controllability. We empirically validate the impact of tilted autoencoder evaluation on controllability by studying several recent ImageNet AEs. Using a multi-dimensional condition-drift evaluation protocol reflecting controllable generation tasks, we find that gFID is only weakly predictive of condition preservation, whereas reconstruction-oriented metrics are substantially more aligned. ControlNet experiments further confirm that controllability tracks condition preservation rather than gFID. Overall, our results expose a gap between ImageNet-centric AE evaluation and the requirements of scalable controllable diffusion, offering practical guidance for more reliable benchmarking and model selection.
Paper Structure (43 sections, 3 theorems, 18 equations, 9 figures, 5 tables)

This paper contains 43 sections, 3 theorems, 18 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Latent Diffusion Models
  4. Improved Autoencoders in Latent Diffusion
  5. Preliminaries
  6. Controllable Latent Diffusion Model
  7. Evaluation Metrics and Trade-offs
  8. Dominance of gFID in AE Evaluation
  9. Reconstruction quality is increasingly under-reported.
  10. Model selection systematically favors generation over reconstruction.
  11. Takeaway.
  12. Autoencoder-Induced Condition Drift
  13. Formulation
  14. Irreducible Condition Drift from Autoencoders
  15. Objective shift induced by the autoencoder.
  16. ...and 28 more sections

Key Result

Theorem 5.2

Fix an autoencoder $(\mathcal{E},\mathcal{D})$ and a condition projector $\phi$. Assume the diffusion backbone perfectly fits the induced latent conditional distribution, i.e., $p_\theta(z\mid c) = p_{\mathcal{E}}(z\mid c)$, and generation follows $G_\theta(c)=\mathcal{D}(z)$ with $z\sim p_\theta(z\

Figures (9)

  • Figure 1: Illustration of how autoencoder affects controllable diffusion generation. We utilize RAE as example, where the controllable generation is realized by trained ControlNet.
  • Figure 2: Spearman correlation (absolute values) matrix across metrics. gFID shows weak correlation with most instance-level and drift measures, while reconstruction metrics align more strongly. To make the contrast more apparent, we restrict the color scale to the range $0.8–1.0$.
  • Figure 3: Common metrics vs. condition drift. gFID shows weak correlation with drift. rFID aligns with drift on average but remains incomplete as a distributional score. Instance-level fidelity correlates strongly with many drift measures and serves as a simple sanity signal.
  • Figure 4: Qualitative controllable generation results. A grid comparing Canny-to-image and Depth-to-image outputs across different frozen autoencoders.
  • Figure 5: Scatter plots between all metrics and gFID.
  • ...and 4 more figures

Theorems & Definitions (5)

  • Definition 5.1: Condition Drift
  • Theorem 5.2: Alignment Limit at Latent Optimum
  • Proposition 5.4: Marginal Matching Does Not Identify Drift
  • Theorem \ref{thm:cc_floor}: Alignment Limit at Latent Optimum
  • proof : Proof of Theorem \ref{['thm:cc_floor']}