Adversarial Diffusion Compression for Real-World Image Super-Resolution

TL;DR

This work tackles Real-ISR under unknown real-world degradations by compressing stable diffusion-based one-step models. The authors introduce AdcSR, a diffusion-GAN hybrid obtained via Adversarial Diffusion Compression (ADC), which removes nonessential modules and prunes the rest of a state-of-the-art one-step model (OSEDiff) and then trains the compact network in two stages: pretraining a pruned VAE decoder and adversarially distilling knowledge from the teacher in the feature space. The approach achieves substantial efficiency gains (e.g., up to $3.7\times$ faster than the teacher and up to $9.3\times$ faster than a competing one-step method) while maintaining competitive Real-ISR recovery quality on synthetic and real datasets. This balance of high-quality reconstruction and real-time inference enables practical deployment on edge and mobile platforms, with code and models released for reproducibility.

Abstract

Real-world image super-resolution (Real-ISR) aims to reconstruct high-resolution images from low-resolution inputs degraded by complex, unknown processes. While many Stable Diffusion (SD)-based Real-ISR methods have achieved remarkable success, their slow, multi-step inference hinders practical deployment. Recent SD-based one-step networks like OSEDiff and S3Diff alleviate this issue but still incur high computational costs due to their reliance on large pretrained SD models. This paper proposes a novel Real-ISR method, AdcSR, by distilling the one-step diffusion network OSEDiff into a streamlined diffusion-GAN model under our Adversarial Diffusion Compression (ADC) framework. We meticulously examine the modules of OSEDiff, categorizing them into two types: (1) Removable (VAE encoder, prompt extractor, text encoder, etc.) and (2) Prunable (denoising UNet and VAE decoder). Since direct removal and pruning can degrade the model's generation capability, we pretrain our pruned VAE decoder to restore its ability to decode images and employ adversarial distillation to compensate for performance loss. This ADC-based diffusion-GAN hybrid design effectively reduces complexity by 73% in inference time, 78% in computation, and 74% in parameters, while preserving the model's generation capability. Experiments manifest that our proposed AdcSR achieves competitive recovery quality on both synthetic and real-world datasets, offering up to 9.3$\times$ speedup over previous one-step diffusion-based methods. Code and models are available at https://github.com/Guaishou74851/AdcSR.

Figures (13)

• Figure 1: Comparison between our proposed AdcSR and typical one-step diffusion-based Real-ISR methods.(a) The state-of-the-art one-step diffusion network OSEDiff wu2024one employs complete SD stabilityai models for Real-ISR, suffering from high computational costs. (b) We distill OSEDiff (ADC-teacher) into a smaller diffusion-GAN hybrid model, AdcSR (ADC-student), under the proposed ADC framework, achieving significantly improved efficiency while maintaining competitive recovery performance.
• Figure 2: Comparison of our proposed AdcSR with other existing one-step diffusion-based Real-ISR methodswang2024sinsrwu2024onezhang2024degradation in terms of visual quality of super-resolution images (top) and model efficiency (bottom). The proposed AdcSR model shows competitive performance in recovering photo-realistic details, while providing the highest inference speed on an NVIDIA A100 GPU, the lowest computational cost, and the second-fewest parameters.
• Figure 3: Illustration of the training and inference processes of AdcSR, an instantiation of our ADC framework applied to OSEDiff.(a) In Stage 1, we pretrain a pruned VAE decoder that shares the latent space with SD and OSEDiff. (b) In Stage 2, we distill the knowledge from OSEDiff (ADC-teacher) into AdcSR (ADC-student) by aligning features in the pretrained decoder. An adversarial loss encourages the student to generate features that can fool a LoRA-finetuned SD UNet (ADC-discriminator), utilizing the corresponding real features of GT images. Since all supervisions perform in the feature space, there is no need to decode images as in previous approaches wu2024onezhang2024degradation. (c) During inference, the LR image is directly fed into our trained compressed UNet and VAE decoder to obtain the super-resolution result.
• Figure 4: Efficiency comparison using a bubble plot, showing the inference time, computation, and parameter number (see Tab. \ref{['tab:comp_diff_quantitative']}) for super-resolving a $128\times 128$ LR image on an NVIDIA A100 GPU. AdcSR achieves the fastest inference, lightest computation, and second-fewest parameters. Bubble colors represent approach types: green for multi-step, blue for one-step, and red for AdcSR.
• Figure 5: Qualitative comparison on images named "0835_pch_00035" from DIV2K-Val (top) and "Nikon_045" from RealSR (bottom).