Q-DiT4SR: Exploration of Detail-Preserving Diffusion Transformer Quantization for Real-World Image Super-Resolution

TL;DR

This work tackles the heavy runtime cost of Diffusion Transformer-based Real-ISR by introducing Q-DiT4SR, a PTQ framework tailored to DiTs. It combines Hierarchical SVD (global low-rank plus local block-wise rank-1) with VaSMP cross-layer weight-bit allocation and VaTMP temporal activation scheduling to preserve high-frequency textures under aggressive quantization. The approach achieves state-of-the-art performance under W4A6 and W4A4, with substantial model size and FLOPs reductions (e.g., 5.8× smaller and over 60× fewer FLOPs at W4A4) while requiring minimal calibration data. This enables practical deployment of DiT-based Real-ISR on resource-constrained devices, and the authors provide code and models for reproducibility.

Abstract

Recently, Diffusion Transformers (DiTs) have emerged in Real-World Image Super-Resolution (Real-ISR) to generate high-quality textures, yet their heavy inference burden hinders real-world deployment. While Post-Training Quantization (PTQ) is a promising solution for acceleration, existing methods in super-resolution mostly focus on U-Net architectures, whereas generic DiT quantization is typically designed for text-to-image tasks. Directly applying these methods to DiT-based super-resolution models leads to severe degradation of local textures. Therefore, we propose Q-DiT4SR, the first PTQ framework specifically tailored for DiT-based Real-ISR. We propose H-SVD, a hierarchical SVD that integrates a global low-rank branch with a local block-wise rank-1 branch under a matched parameter budget. We further propose Variance-aware Spatio-Temporal Mixed Precision: VaSMP allocates cross-layer weight bit-widths in a data-free manner based on rate-distortion theory, while VaTMP schedules intra-layer activation precision across diffusion timesteps via dynamic programming (DP) with minimal calibration. Experiments on multiple real-world datasets demonstrate that our Q-DiT4SR achieves SOTA performance under both W4A6 and W4A4 settings. Notably, the W4A4 quantization configuration reduces model size by 5.8$\times$ and computational operations by over 60$\times$. Our code and models will be available at https://github.com/xunzhang1128/Q-DiT4SR.

Figures (9)

• Figure 1: Visual comparison ($\times$4) under W4A4 setting. Compared with existing quantization methods, our quantized 4-bit Q-DiT4SR better preserves fine-grained textures, while remaining visually close to the full-precision (FP) model. We also provide the parameter (i.e., Params (M)) and operation numbers (i.e., Ops (G)).
• Figure 2: Left: Visual comparisons ($\times4$) on RealLR200 under W4A6. Right: Frequency energy of the blocks.23.ff.net.2 layer outputs from three W4A6 models, compared with the FP model.
• Figure 3: Overview of our proposed Q-DiT4SR framework. (a) Forward with H-SVD: we reconstruct and quantize DiT weights using a hierarchical decomposition that integrates SVD-G with SVD-L. (b) VaSMP: a cross-layer mixed-precision assignment for weights driven by mean weight variance. (c) VaTMP: a intra-layer scheduling that allocates activation precision across diffusion timesteps using mean activation variance. The spatio-temporal grid specifies per-(layer, timestep) bit configurations (weight/activation bit pairs) for deployment.
• Figure 4: PCA analysis on RealSR Canon_001. Left: Top-128 principal components (PCs) are progressively removed from the output of the blocks.23.ff.net.2 layer in the FP model. Removing dominant PCs leads to a significant degradation in SR quality, highlighting their importance in SR tasks. Right: Comparison between SVD-G and H-SVD under matched parameter budgets for the same non-quantized layer, with outputs projected onto the FP PC space. The hierarchical decomposition aligns more closely with the FP model, demonstrating the effectiveness of H-SVD.
• Figure 5: Variance Distribution Analysis. The boxplot is generated by randomly selecting 32 different types of layers and calculating the variance of 128 randomly selected output channels for each layer. The variance of Hadamard-transformed weights varies by orders of magnitude across layers, while remaining relatively stable across output channels within a single layer.