Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning

TL;DR

Edge-SD-SR tackles the challenge of running diffusion-based super-resolution on edge devices by introducing a compact ~169M-parameter model with ~142 GFLOPs and a threefold training strategy: bidirectional conditioning on the LR input, joint encoder-UNet training with decoupled HR/LR encodings, and decoder fine-tuning within a 1-step, scale-distillation framework. The method achieves real-time on-device SR (e.g., 128×128 → 512×512 in ~38 ms on a Samsung S24, and 512×512 → 2048×2048 in ~1.1 s across 25 evaluations) while matching or surpassing larger diffusion-based SR baselines on standard benchmarks such as DIV2K, RealSR, and DRealSR. Thorough ablations confirm the critical roles of bidirectional conditioning, variance conditioning, and joint encoder training in preserving quality at a fraction of the compute, enabling practical on-device SR for mobile devices.

Abstract

There has been immense progress recently in the visual quality of Stable Diffusion-based Super Resolution (SD-SR). However, deploying large diffusion models on computationally restricted devices such as mobile phones remains impractical due to the large model size and high latency. This is compounded for SR as it often operates at high res (e.g. 4Kx3K). In this work, we introduce Edge-SD-SR, the first parameter efficient and low latency diffusion model for image super-resolution. Edge-SD-SR consists of ~169M parameters, including UNet, encoder and decoder, and has a complexity of only ~142 GFLOPs. To maintain a high visual quality on such low compute budget, we introduce a number of training strategies: (i) A novel conditioning mechanism on the low resolution input, coined bidirectional conditioning, which tailors the SD model for the SR task. (ii) Joint training of the UNet and encoder, while decoupling the encodings of the HR and LR images and using a dedicated schedule. (iii) Finetuning the decoder using the UNet's output to directly tailor the decoder to the latents obtained at inference time. Edge-SD-SR runs efficiently on device, e.g. it can upscale a 128x128 patch to 512x512 in 38 msec while running on a Samsung S24 DSP, and of a 512x512 to 2048x2048 (requiring 25 model evaluations) in just ~1.1 sec. Furthermore, we show that Edge-SD-SR matches or even outperforms state-of-the-art SR approaches on the most established SR benchmarks.

Figures (5)

• Figure 1: Results of $4\times$ upsampling with (a) interpolation, (b) our architecture with the standard training approach (c) our architecture with our training approach. Images are best seen in a display and zoomed in. Samples taken from DIV2K-RealESRGAN dataset.
• Figure 2: Qualitative results. (a) input LR, (b) StableSR 200-steps (c) Resshift 15-Steps (d) Ours 1-step (e) ground truth. Samples are taken from DRealSR dataset. The images are best seen in a display and zoomed in. ResShift and StableSR include $173M$ and $1043M$ parameters and both require $> 2,000$ GFLOPs to process a $128\times128$ patch, respectively. Our model includes only 169M parameters and requires only 142 GFLOPs. Despite being hugely cost and parameter-efficient, our method works competitively with the larger baselines.
• Figure 3: Overview of bidirectional conditioning for training. We use different encoders for HR and LR, where $E_{HR}$ is frozen. We use the LR embeddings to condition the Gaussian mean and variance. The U-Net takes only $\mathbf{z}_t$ as input and is trained jointly with the encoder.
• Figure 4: Qualitative results on samples from the DIV2K-RealESRGAN dataset. (a) input LR, (b) StableSR 200-steps (c) Resshift 15-Steps (d) Ours 1-step (e) ground truth. ResShift and StableSR include $173M$ and $1043M$ parameters and both require $> 2,000$ GFLOPs to process a $128\times128$ patch, resp. Ours includes only 169M parameters and requires 142 GFLOPs. Despite being hugely cost and parameter-efficient, our method works competitively with the larger baselines.
• Figure 5: Qualitative results on samples from the RealSR and DRealSR datasets capturing real camera degradations. (a) input LR, (b) StableSR 200-steps (c) Resshift 15-Steps (d) Ours 1-step (e) ground truth. ResShift and StableSR include $173M$ and $1043M$ parameters and both require $> 2,000$ GFLOPs to process a $128\times128$ patch, resp. Ours includes only 169M parameters and requires 142 GFLOPs. Despite being hugely cost and parameter-efficient, our method works competitively with the larger baselines.