Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TMPDiff: Temporal Mixed-Precision for Diffusion Models

Basile Lewandowski, Simon Kurz, Aditya Shankar, Robert Birke, Jian-Jia Chen, Lydia Y. Chen

Abstract

Diffusion models are the go-to method for Text-to-Image generation, but their iterative denoising processes has high inference latency. Quantization reduces compute time by using lower bitwidths, but applies a fixed precision across all denoising timesteps, leaving an entire optimization axis unexplored. We propose TMPDiff, a temporal mixed-precision framework for diffusion models that assigns different numeric precision to different denoising timesteps. We hypothesize that quantization errors accumulate additively across timesteps, which we then validate experimentally. Based on our observations, we develop an adaptive bisectioning-based algorithm, which assigns per-step precisions with linear evaluation complexity, reducing an otherwise exponential search problem. Across four state-of-the-art diffusion models and three datasets, TMPDiff consistently outperforms uniform-precision baselines at matched speedup, achieving 10 to 20% improvement in perceptual quality. On FLUX.1-dev, TMPDiff achieves 90% SSIM relative to the full-precision model at a speedup of 2.5x over 16-bit inference.

TMPDiff: Temporal Mixed-Precision for Diffusion Models

Abstract

Diffusion models are the go-to method for Text-to-Image generation, but their iterative denoising processes has high inference latency. Quantization reduces compute time by using lower bitwidths, but applies a fixed precision across all denoising timesteps, leaving an entire optimization axis unexplored. We propose TMPDiff, a temporal mixed-precision framework for diffusion models that assigns different numeric precision to different denoising timesteps. We hypothesize that quantization errors accumulate additively across timesteps, which we then validate experimentally. Based on our observations, we develop an adaptive bisectioning-based algorithm, which assigns per-step precisions with linear evaluation complexity, reducing an otherwise exponential search problem. Across four state-of-the-art diffusion models and three datasets, TMPDiff consistently outperforms uniform-precision baselines at matched speedup, achieving 10 to 20% improvement in perceptual quality. On FLUX.1-dev, TMPDiff achieves 90% SSIM relative to the full-precision model at a speedup of 2.5x over 16-bit inference.
Paper Structure (24 sections, 16 equations, 11 figures, 5 tables, 1 algorithm)

This paper contains 24 sections, 16 equations, 11 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Problem Definition
  3. Additive Error Model and Empirical Analysis
  4. Additive Error Ansatz for Mixed Precision Schedules
  5. Validation 1: Single Timestep Error Contributions
  6. Validation 2: Multistep Error Contributions
  7. TMPDiff: A Mixed-Precision Scheduling Framework
  8. Targeting High Speedups
  9. Temporal Search Space Exploration
  10. Adaptive Bisection Algorithm for Precision Scheduling
  11. Experimental Results
  12. Experimental Setup
  13. Pareto Optimality
  14. Image Quality
  15. Mix of Models
  16. ...and 9 more sections

Figures (11)

  • Figure 1: Change in final latent error $E(Z)$ from single-timestep up- and downcasting.
  • Figure 2: Linear fit between the measured final latent error $E(Z)$ and the schedule scoring functions $S^{\uparrow}(Z)$ and $S^{\downarrow}(Z)$ for upcasting $K$ timesteps for Sana.
  • Figure 3: Rank deviation per $K$ over the number of sampled timesteps for calibration on the COCO dataset for the Sana model.
  • Figure 4: Performance metrics over different speedup factors on FLUX W4A4R32.
  • Figure 5: Linear fit between ranking scores $S^{\uparrow}(Z)$ and measured ground truth variance for the Sana model on new COCO samples.
  • ...and 6 more figures