Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization

Jinchao Zhu, Yuxuan Wang, Xiaobing Tu, Siyuan Pan, Pengfei Wan, Gao Huang

TL;DR

This work first explores the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy, and proposes a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.

Abstract

The Stable Diffusion Model (SDM) is a popular and efficient text-to-image (t2i) generation and image-to-image (i2i) generation model. Although there have been some attempts to reduce sampling steps, model distillation, and network quantization, these previous methods generally retain the original network architecture. Billion scale parameters and high computing requirements make the research of model architecture adjustment scarce. In this work, we first explore the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy. Secondly, in order to maintaining the model performance, we add cross-layer multi-expert conditional convolution (CLME-Condconv) to the block pruning part to inherit the original convolution parameters. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part. Finally, we use semantic-aware supervision (SAS) to align the outputs of the teacher model and student model at the semantic level. Experiments show that this method can effectively train a lightweight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. Experiments show that the proposed method can effectively train a light-weight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. After acceleration, the UNet part of the model is 22% faster and the overall speed is 19% faster.

A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization

TL;DR

This work first explores the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy, and proposes a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.

Abstract

The Stable Diffusion Model (SDM) is a popular and efficient text-to-image (t2i) generation and image-to-image (i2i) generation model. Although there have been some attempts to reduce sampling steps, model distillation, and network quantization, these previous methods generally retain the original network architecture. Billion scale parameters and high computing requirements make the research of model architecture adjustment scarce. In this work, we first explore the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy. Secondly, in order to maintaining the model performance, we add cross-layer multi-expert conditional convolution (CLME-Condconv) to the block pruning part to inherit the original convolution parameters. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part. Finally, we use semantic-aware supervision (SAS) to align the outputs of the teacher model and student model at the semantic level. Experiments show that this method can effectively train a lightweight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. Experiments show that the proposed method can effectively train a light-weight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. After acceleration, the UNet part of the model is 22% faster and the overall speed is 19% faster.
Paper Structure (12 sections, 6 figures, 4 tables)

This paper contains 12 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Compression Methed
  3. UNet architecture
  4. Progressive Incubation Strategy
  5. Cross-layer Multi-expert Conditional Convolution
  6. Multi-UNet Dynamic Processing
  7. Semantic-Aware Supervision
  8. Experiments
  9. Datasets
  10. Implementation
  11. Qualitative evaluation.
  12. Conclusion

Figures (6)

  • Figure 1: Parameters and computation of Stable Diffusion v1 and Stable Diffusion v2.
  • Figure 2: Latency (iPhone 14 Pro, ms) and parameter (M) analysis for cross-attention (CA) and ResNet blocks in the UNet of Stable Diffusion.
  • Figure 3: Macro progressive incubation Process.
  • Figure 4: Specific stable diffusion progressive incubation process.
  • Figure 5: Cross-layer multi-expert conditional convolution.
  • ...and 1 more figures