Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation

TL;DR

This work provides a flow-map–based theoretical analysis of consistency models, identifying instability and suboptimal convergence when training from scratch. It introduces time-condition relaxation and a reformulated self-distillation (iSD) to stabilize optimization and improve reproducibility, including a training-time CFG variant (iSD-T) that removes reliance on pretrained initializers. Empirically, iSD achieves competitive or superior results on ImageNet-1K and CelebA-HQ with reduced variance, and extends to diffusion-based policy learning, demonstrating broader applicability beyond image generation. Theoretically, it connects direct training, Eulerian distillation, and consistency objectives through a unified flow-map view, clarifying when fixed points arise and how marginal velocity guidance mitigates degeneracy, with practical implications for robust, reproducible generative modeling.

Abstract

Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models. However, these methods exhibit inherent instability and limited reproducibility when training from scratch, motivating subsequent work to explain and stabilize these issues. While these efforts have provided valuable insights, the explanations remain fragmented, and the theoretical relationships remain unclear. In this work, we provide a theoretical examination of consistency models by analyzing them from a flow map-based perspective. This joint analysis clarifies how training stability and convergence behavior can give rise to degenerate solutions. Building on these insights, we revisit self-distillation as a practical remedy for certain forms of suboptimal convergence and reformulate it to avoid excessive gradient norms for stable optimization. We further demonstrate that our strategy extends beyond image generation to diffusion-based policy learning, without reliance on a pretrained diffusion model for initialization, thereby illustrating its broader applicability.

Figures (16)

• Figure 1: From consistency training to improved Self-Distillation. Consistency training learns a mapping over conditional velocity, often suffering from training instability and reproducibility issues. Relaxing the time condition mitigates this instability, and self-distillation provides a principled target by aligning with the marginal velocity. However, directly applying self-distillation leads to unstable training. We therefore reformulate the objective and incorporate classifier-free guidance, further stabilizing training and improving reproducibility.
• Figure 2: Toy experiments with a 5-layer MLP (batch size of 2048). $\mathcal{L}_\mathrm{DT}$ drives flow map training toward a suboptimal solution, while $\mathcal{L}_\mathrm{ED}$ leads to a solution close to the ground-truth.
• Figure 3: Consistency training (with $\mathcal{L}_\mathrm{CT}$) is biased toward degenerate distributions when the batch size $B$ decreases.
• Figure 4: $\mathcal{L}_\mathrm{ED}$ over training steps on a toy dataset. The experiment follows \ref{['fig:toy']}, except for the batch size (solid lines indicate a batch size of 2048, and dash-dot lines indicate 128).
• Figure 5: Loss landscapes of four methods. $\alpha$ and $\beta$ denote the top-2 eigenvectors of the Hessian on ImageNet-1K with DiT-B/4. $\sigma$ denotes the standard deviations of each landscape, and $N$ the number of samples outside each method's own 95% confidence bound. Values in parentheses report the number of samples exceeding the 95% bound defined by iSD, as a common reference (details in \ref{['appendix:loss-landscape']}).

Theorems & Definitions (3)

• Proposition 4.1
• Proposition 4.2
• Proposition 4.3