RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation

TL;DR

RepLDM tackles the challenge of producing high-resolution images with pretrained latent diffusion models by reprogramming them without retraining. It decomposes denoising into a training-free attention-guided stage to improve latent structure, followed by a pixel-space progressive upsampling stage that avoids latent-space artifacts and enables faster HR generation. The approach achieves state-of-the-art quality and efficiency across multiple resolutions, with a reported 5x speedup and strong qualitative results, validated by quantitative metrics and human studies. Its practical impact lies in enabling accessible, high-quality HR image generation on consumer hardware without costly retraining.

Abstract

While latent diffusion models (LDMs), such as Stable Diffusion, are designed for high-resolution (HR) image generation, they often struggle with significant structural distortions when generating images at resolutions higher than their training one. Instead of relying on extensive retraining, a more resource-efficient approach is to reprogram the pretrained model for HR image generation; however, existing methods often result in poor image quality and long inference time. We introduce RepLDM, a novel reprogramming framework for pretrained LDMs that enables high-quality, high-efficiency, high-resolution image generation; see Fig. 1. RepLDM consists of two stages: (i) an attention guidance stage, which generates a latent representation of a higher-quality training-resolution image using a novel training-free self-attention mechanism to enhance the structural consistency; and (ii) a progressive upsampling stage, which progressively performs upsampling in pixel space to mitigate the severe artifacts caused by latent space upsampling. The effective initialization from the first stage allows for denoising at higher resolutions with significantly fewer steps, improving the efficiency. Extensive experimental results demonstrate that RepLDM significantly outperforms state-of-the-art methods in both quality and efficiency for HR image generation, underscoring its advantages for real-world applications. Codes: https://github.com/kmittle/RepLDM.

Figures (27)

• Figure 1: High-resolution images generated by our RepLDM using a single consumer-grade 3090 GPU. The corresponding thumbnails are generated by SDXL sdxl at their training resolution.
• Figure 2: Comparison of our RepLDM with prior work in generating 2048$\times$2048 image. The prompt is Neon lights illuminate the bustling cityscape at night, casting colorful reflections on the wet streets. Zoom-in for a better view.
• Figure 3: Overview of RepLDM. RepLDM divides the denoising process of a pre-trained LDM into two stages. The first stage leverages the introduced attention guidance to enhance the structural consistency by utilizing a novel training-free self-attention mechanism (TFSA). The second stage iteratively upsamples the latent representation in pixel space to eliminate artifacts.
• Figure 4: Comparison between upsampling in pixel space and latent space. (a) RepLDM with latent space upsampling leads to severe artifacts. (b)-(e): Qualitative and quantitative comparisons of different upsampling methods.
• Figure 5: Qualitative comparison with other baselines. The prompts used to generate the images are presented in the white boxes. MultiDiffusion fails to maintain global semantic consistency. DemoFusion and AccDiffusion exhibit severe artifacts and content repetition. The red boxes indicate some synthesis errors. Zoom-in for a better view.