Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing

Zhenyuan Dong, Sai Qian Zhang

TL;DR

DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference, relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth.

Abstract

Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation, surpassing the performance of traditional diffusion models that employ U-Net. However, the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs, which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT, we adopt the layer-wise grid search strategy to optimize the smoothing factor. Moreover, we integrate a training-free LoRA module for weight quantization, leveraging alternating optimization to minimize quantization errors without additional fine-tuning. Experimental results demonstrate that our approach enables 4-bit weight, 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model.

DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing

TL;DR

DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference, relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth.

Abstract

Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation, surpassing the performance of traditional diffusion models that employ U-Net. However, the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs, which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT, we adopt the layer-wise grid search strategy to optimize the smoothing factor. Moreover, we integrate a training-free LoRA module for weight quantization, leveraging alternating optimization to minimize quantization errors without additional fine-tuning. Experimental results demonstrate that our approach enables 4-bit weight, 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model.
Paper Structure (23 sections, 10 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 23 sections, 10 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Models Quantization
  4. Diffusion Transformer Quantization
  5. Methodology
  6. Preliminaries
  7. Diffusion Transformers.
  8. Asymmetric INT Quantization.
  9. Temporal-aggregated Smoothing
  10. Grid Search Optimization
  11. Activation and Weight Quantization
  12. Fine-Grained Weight Quantization
  13. Alternating Optimization for LoRA Integration
  14. Experiments
  15. Experiment Settings
  16. ...and 8 more sections

Figures (9)

  • Figure 1: DiTAS architecture.
  • Figure 2: Input activation range across different time steps. The dark blue segment shows the 95th percentile range, the light blue segment denotes the extreme values.
  • Figure 3: Activation range before Temporal-aggregated Smoothing (TAS).
  • Figure 4: Activation range after Temporal-aggregated Smoothing (TAS).
  • Figure 5: Weight with outliers across input channels in 7th DiT Block's QKV layer.
  • ...and 4 more figures