Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Model Compression for Image-to-Image Translation

Geonung Kim, Beomsu Kim, Eunhyeok Park, Sunghyun Cho

TL;DR

This work tackles the practical bottlenecks of diffusion-based image-to-image translation by introducing a task-aware compression strategy that combines depth-skip pruning and time-step optimization. By pruning non-critical deep U-Net blocks and aggressively reordering denoising steps with a gamma-based schedule, the method achieves substantial reductions in parameter count and latency across IP2P, StableSR, and ControlNet while preserving output quality. It offers a training-free alternative to heavy pruning and step-distillation approaches, outperforming prior methods in both speed and perceptual metrics. The approach is particularly impactful for real-world I2I applications where compute and latency are critical, and it demonstrates broad applicability, including compatibility with multiple schedulers and downstream tasks.

Abstract

As recent advances in large-scale Text-to-Image (T2I) diffusion models have yielded remarkable high-quality image generation, diverse downstream Image-to-Image (I2I) applications have emerged. Despite the impressive results achieved by these I2I models, their practical utility is hampered by their large model size and the computational burden of the iterative denoising process. In this paper, we propose a novel compression method tailored for diffusion-based I2I models. Based on the observations that the image conditions of I2I models already provide rich information on image structures, and that the time steps with a larger impact tend to be biased, we develop surprisingly simple yet effective approaches for reducing the model size and latency. We validate the effectiveness of our method on three representative I2I tasks: InstructPix2Pix for image editing, StableSR for image restoration, and ControlNet for image-conditional image generation. Our approach achieves satisfactory output quality with 39.2%, 56.4% and 39.2% reduction in model footprint, as well as 81.4%, 68.7% and 31.1% decrease in latency to InstructPix2Pix, StableSR and ControlNet, respectively.

Diffusion Model Compression for Image-to-Image Translation

TL;DR

This work tackles the practical bottlenecks of diffusion-based image-to-image translation by introducing a task-aware compression strategy that combines depth-skip pruning and time-step optimization. By pruning non-critical deep U-Net blocks and aggressively reordering denoising steps with a gamma-based schedule, the method achieves substantial reductions in parameter count and latency across IP2P, StableSR, and ControlNet while preserving output quality. It offers a training-free alternative to heavy pruning and step-distillation approaches, outperforming prior methods in both speed and perceptual metrics. The approach is particularly impactful for real-world I2I applications where compute and latency are critical, and it demonstrates broad applicability, including compatibility with multiple schedulers and downstream tasks.

Abstract

As recent advances in large-scale Text-to-Image (T2I) diffusion models have yielded remarkable high-quality image generation, diverse downstream Image-to-Image (I2I) applications have emerged. Despite the impressive results achieved by these I2I models, their practical utility is hampered by their large model size and the computational burden of the iterative denoising process. In this paper, we propose a novel compression method tailored for diffusion-based I2I models. Based on the observations that the image conditions of I2I models already provide rich information on image structures, and that the time steps with a larger impact tend to be biased, we develop surprisingly simple yet effective approaches for reducing the model size and latency. We validate the effectiveness of our method on three representative I2I tasks: InstructPix2Pix for image editing, StableSR for image restoration, and ControlNet for image-conditional image generation. Our approach achieves satisfactory output quality with 39.2%, 56.4% and 39.2% reduction in model footprint, as well as 81.4%, 68.7% and 31.1% decrease in latency to InstructPix2Pix, StableSR and ControlNet, respectively.
Paper Structure (35 sections, 3 equations, 15 figures, 13 tables, 2 algorithms)

This paper contains 35 sections, 3 equations, 15 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. I2I Downstream Tasks based on T2I Diffusion Models
  4. Model Pruning of Diffusion Models
  5. Acceleration of Diffusion Models
  6. Methods
  7. Diffusion Models
  8. Depth-Skip Pruning
  9. Single- vs. multi-depth search
  10. Optimizing quality or latency while fixing model size
  11. Optimizing model size or quality while fixing latency
  12. Optimizing model size or latency while fixing quality
  13. Time-step Optimization
  14. Experiments
  15. Qualitative and Quantitative Comparisons
  16. ...and 20 more sections

Figures (15)

  • Figure 1: Motivations of our approach. (a) Even after removing the network layers beneath a certain depth, IP2P ip2p, a downstream I2I model, still produces a plausible result. (b) By focusing on earlier time steps, a feasible output can be obtained using only five denoising steps.
  • Figure 2: (a) Depth-skip pruning eliminates all layers deeper than a certain depth level, effectively reducing the model size. (b) Given a fixed number of time steps, our time-step optimization finds differently biased time step sequences for different I2I tasks.
  • Figure 3: Qualitative examples of our depth-skip pruning and time-step optimization on IP2P ip2p, StableSR stablesr, and ControlNet controlnet.
  • Figure 4: Comparison of the depth-skip pruning and previous pruning methods. The number on the top-right side in each image denotes the pruned model size.
  • Figure 5: (a) Quantitative comparison of the depth-skip pruning and the other methods on IP2P ip2p. "T.O." denotes the time-step optimization. (b-d) Comparison between the time-step optimization and uniform sampling.
  • ...and 10 more figures