Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling

Qinsheng Zhang, Jiaming Song, Yongxin Chen

TL;DR

This work addresses slow diffusion-model sampling by reexamining high-order exponential-integrator solvers and identifying missing order conditions that cause degradation. It introduces Refined Exponential Solver (RES), a unified, order-condition–aware approach that leverages multiple probability-flow ODE parameterizations (notably logSNR/negative logSNR) to achieve smoother dynamics and lower discretization error. The authors derive and enforce order conditions for single-step schemes, extend the method to stochastic and multi-step samplers, and demonstrate substantial practical gains, including up to 60% reductions in numerical defects and meaningful FID improvements under suboptimal time schedules and across guided and cascaded models. Overall, RES provides a robust, generalizable improvement to diffusion-model sampling that translates to faster, more reliable image generation without retraining the model.

Abstract

Efficient differential equation solvers have significantly reduced the sampling time of diffusion models (DMs) while retaining high sampling quality. Among these solvers, exponential integrators (EI) have gained prominence by demonstrating state-of-the-art performance. However, existing high-order EI-based sampling algorithms rely on degenerate EI solvers, resulting in inferior error bounds and reduced accuracy in contrast to the theoretically anticipated results under optimal settings. This situation makes the sampling quality extremely vulnerable to seemingly innocuous design choices such as timestep schedules. For example, an inefficient timestep scheduler might necessitate twice the number of steps to achieve a quality comparable to that obtained through carefully optimized timesteps. To address this issue, we reevaluate the design of high-order differential solvers for DMs. Through a thorough order analysis, we reveal that the degeneration of existing high-order EI solvers can be attributed to the absence of essential order conditions. By reformulating the differential equations in DMs and capitalizing on the theory of exponential integrators, we propose refined EI solvers that fulfill all the order conditions, which we designate as Refined Exponential Solver (RES). Utilizing these improved solvers, RES exhibits more favorable error bounds theoretically and achieves superior sampling efficiency and stability in practical applications. For instance, a simple switch from the single-step DPM-Solver++ to our order-satisfied RES solver when Number of Function Evaluations (NFE) $=9$, results in a reduction of numerical defects by $25.2\%$ and FID improvement of $25.4\%$ (16.77 vs 12.51) on a pre-trained ImageNet diffusion model.

Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling

TL;DR

This work addresses slow diffusion-model sampling by reexamining high-order exponential-integrator solvers and identifying missing order conditions that cause degradation. It introduces Refined Exponential Solver (RES), a unified, order-condition–aware approach that leverages multiple probability-flow ODE parameterizations (notably logSNR/negative logSNR) to achieve smoother dynamics and lower discretization error. The authors derive and enforce order conditions for single-step schemes, extend the method to stochastic and multi-step samplers, and demonstrate substantial practical gains, including up to 60% reductions in numerical defects and meaningful FID improvements under suboptimal time schedules and across guided and cascaded models. Overall, RES provides a robust, generalizable improvement to diffusion-model sampling that translates to faster, more reliable image generation without retraining the model.

Abstract

Efficient differential equation solvers have significantly reduced the sampling time of diffusion models (DMs) while retaining high sampling quality. Among these solvers, exponential integrators (EI) have gained prominence by demonstrating state-of-the-art performance. However, existing high-order EI-based sampling algorithms rely on degenerate EI solvers, resulting in inferior error bounds and reduced accuracy in contrast to the theoretically anticipated results under optimal settings. This situation makes the sampling quality extremely vulnerable to seemingly innocuous design choices such as timestep schedules. For example, an inefficient timestep scheduler might necessitate twice the number of steps to achieve a quality comparable to that obtained through carefully optimized timesteps. To address this issue, we reevaluate the design of high-order differential solvers for DMs. Through a thorough order analysis, we reveal that the degeneration of existing high-order EI solvers can be attributed to the absence of essential order conditions. By reformulating the differential equations in DMs and capitalizing on the theory of exponential integrators, we propose refined EI solvers that fulfill all the order conditions, which we designate as Refined Exponential Solver (RES). Utilizing these improved solvers, RES exhibits more favorable error bounds theoretically and achieves superior sampling efficiency and stability in practical applications. For instance, a simple switch from the single-step DPM-Solver++ to our order-satisfied RES solver when Number of Function Evaluations (NFE) , results in a reduction of numerical defects by and FID improvement of (16.77 vs 12.51) on a pre-trained ImageNet diffusion model.
Paper Structure (40 sections, 5 theorems, 51 equations, 14 figures, 6 tables, 4 algorithms)

This paper contains 40 sections, 5 theorems, 51 equations, 14 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Refined exponential solver RES
  4. Better parameterizations for probability flow ODE
  5. EDM
  6. DEIS
  7. logSNR
  8. Negative logSNR
  9. Single-step numerical schemes and defect analysis
  10. Order conditions
  11. Single-step update for deterministic and stochastic sampler
  12. Multi-step scheme
  13. Experiments
  14. Numerical defects for deterministic sampler
  15. Numerical defects and FID
  16. ...and 25 more sections

Key Result

Lemma 1

Let $\phi_{j}(-h_n) = \frac{1}{h_n^j} \int_{0}^{h_n} e^{\tau - h_n} \frac{\tau^{j-1}}{(j -1)!} d \tau$, the order-$q$ Taylor expansion of the intermediate defects $\delta_{n{,}i}, \delta_{n+1}$ can be formulated as follows: where function $\psi_{j,i}, \psi_j$ for the coefficients of the $(j-1)$-th order derivative of $f(\lambda_n)$ are defined as

Figures (14)

  • Figure 1: \ref{['fig:function-derivative']}. Along the trajectory of an exact solution to the probability flow ODE, semi-linear ODE with logarithm transformation parametrization on noise level $\sigma$ has smaller "acceleration", i.e., the time-derivative of "velocity". The curves are averaged over $512$ solutions on the pre-trained ImageNet model karras2022elucidating. This suggests that semi-linear ODE incur less discretization error when solving ODEs. \ref{['fig:defect']}. The auxiliary trajectory $\hat{{\bm{x}}}$ starts from exact solution ${\bm{x}}$ and is built upon numerical integration proposed in \ref{['eq:sin-scheme']}. The diagram depicts that numerical defects $\Delta_{n+1}$ between $\textcolor{red}{{{\bm{x}}}(\lambda_{n+1})}$ and numerical solution $\textcolor{blue}{\bm{x}}_{n+1}$ is a result of accumulated intermediate defects $\textcolor{black!50!green}{\{\delta_n, \delta_{n,i}\}}$ in each step, which is the discrepancy between the auxiliary trajectory and the exact solution trajectory.
  • Figure 2: Comparison among various deterministic samplers on pretrained ImageNet diffusion model karras2022elucidating. $(\text{S})$ indicates single-step methods while $(\text{M})$ for multistep methods. (Left) Numerical defects of various sampling algorithms vs the number of function evaluation (NFE) with recommended time scheduling karras2022elucidating. (Middle) FID of various sampling algorithms vs NFE with recommended time scheduling. (Right) Numerical defects vs NFE with suboptimal time schedule. RES$(\text{S})$ shows better robustness against suboptimal-scheduling. Remarkably, with only 59 NFE, the single-step RES attains a numerical accuracy on par with the 99 NFE DPM-Solver++ (S).
  • Figure 3: Comparison on DDPM-like classifier-guided diffusion models dhariwal2021diffusion with different guidance.
  • Figure 4: Our efficient single-step method can improve stochastic sampling as shown in \ref{['fig:exp_stochastic_sampler']}. RES outperforms DPM-Solver++ in pixel space text-to-image diffusion models.
  • Figure 5: (Left) The evolution of nonlinear function evaluation for \ref{['eq:eps-ode']} characterized by $f_\epsilon (\sigma) := \epsilon_\theta({\bm{x}}(\sigma), \sigma)$. (Right) The evolution of nonlinear function evaluation for \ref{['eq:semi_ode_eps']}, defined by $f_\epsilon(\lambda) := \epsilon_\theta({\bm{x}}(\lambda), \lambda)$ and $f_D(\lambda) := D_\theta({\bm{x}}(\lambda), \lambda)$. Implementing a logarithmic transformation on the noise level $\sigma$ results in smoother trajectories of nonlinear function evaluations along ODE solutions.
  • ...and 9 more figures

Theorems & Definitions (5)

  • Lemma 1
  • Theorem 1: Error bound for solvers that satisfy the 1st-order condition
  • Theorem 2: Error bound for solvers that satisfy the 1st- and 2nd-order conditions
  • Proposition 1: (Informal)
  • Lemma 2: Discrete Gronwall Inequality