Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion

Samarth N Ramesh, Zhixue Zhao

TL;DR

This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain and uncovers intriguing questions about information encoding in text-to-image models.

Abstract

As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems

Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion

TL;DR

This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain and uncovers intriguing questions about information encoding in text-to-image models.

Abstract

As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems

Paper Structure

This paper contains 30 sections, 2 equations, 14 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Models
  4. Pruning
  5. Magnitude Pruning
  6. SparseGPT
  7. Wanda
  8. Outlier-Weighted Layerwise Pruning (OWL)
  9. Experimental Setup
  10. Pruning Single Component
  11. Full Model Pruning
  12. Dataset
  13. Evaluation
  14. FID
  15. CLIP Score
  16. ...and 15 more sections

Figures (14)

  • Figure 1: Pruning only Text Encoder of Stable Diffusion 2 using Magnitude Pruning - Sharp drop off in performance observed at 62.5%
  • Figure 2: Text Encoder Magnitude Pruning Examples - Text Encoder breaks down beyond 62.5%
  • Figure 3: Pruning only Text Encoder of Stable Diffusion 2 using Wanda Pruning - Sharp drop off in performance observed at 60%
  • Figure 4: Text Encoder Wanda Pruning Examples - Text Encoder breaks down beyond 62.5%
  • Figure 5: Pruning only Text Encoder of Stable Diffusion 2 using Magnitude Pruning with OWL - Sharp drop off in performance observed at 62.5%
  • ...and 9 more figures