A Unified Module for Accelerating STABLE-DIFFUSION: LCM-LORA

TL;DR

The paper tackles slow inference in latent diffusion models by introducing LCM‑LoRA, a training‑free, LoRA‑based accelerator that plugs into fine‑tuned Stable‑Diffusion models to realize fast Latent Consistency Model (LCM) inference. Building on Latent Consistency Models and LoRA, it distills the teacher diffusion model into lightweight adapters that enable high‑fidelity image synthesis with as few as 1–4 steps, and it generalizes across SD variants such as SD‑V1.5, SDXL, and SD‑1B. Empirical results on the LAION‑5B‑Aesthetics dataset show that LCM‑LoRA achieves competitive or superior FID and LPIPS with much fewer steps compared to DDIM, DPM‑Solver, and DPM‑Solver++ while reducing memory footprint, demonstrating strong cross‑model applicability and on‑device potential. The work also discusses limitations, including dependence on pre‑trained LDMs and latent space assumptions, and identifies directions for improving stability and cross‑domain generalization.

Abstract

This paper presents a comprehensive study on the unified module for accelerating stable-diffusion processes, specifically focusing on the lcm-lora module. Stable-diffusion processes play a crucial role in various scientific and engineering domains, and their acceleration is of paramount importance for efficient computational performance. The standard iterative procedures for solving fixed-source discrete ordinates problems often exhibit slow convergence, particularly in optically thick scenarios. To address this challenge, unconditionally stable diffusion-acceleration methods have been developed, aiming to enhance the computational efficiency of transport equations and discrete ordinates problems. This study delves into the theoretical foundations and numerical results of unconditionally stable diffusion synthetic acceleration methods, providing insights into their stability and performance for model discrete ordinates problems. Furthermore, the paper explores recent advancements in diffusion model acceleration, including on device acceleration of large diffusion models via gpu aware optimizations, highlighting the potential for significantly improved inference latency. The results and analyses in this study provide important insights into stable diffusion processes and have important ramifications for the creation and application of acceleration methods specifically, the lcm-lora module in a variety of computing environments.

Figures (3)

• Figure 1: High-resolution image generation with LCM-LoRA. We use LCM-LoRA to distill different pretrained diffusion models and generate images at 512×512 (LCM-LoRA-SD-V1.5) and 1024×1024 (LCM-LoRA-SDXL and LCM-LoRA-SSD-1B) resolutions. We set the classifier-free guidance scale $\omega$ to 8 for all models and obtain all images with only 4 inference steps.
• Figure 2: LCM-LoRA applies LoRA distillation to LCM to reduce the memory consumption and enable the training of larger Stable-Diffusion models, such as SDXL and SSD-1B, with limited resources. LCM-LoRA also allows the seamless integration of different LoRA parameters (‘acceleration vector’ and ‘style vector’) that are obtained from LCM distillation and style fine-tuning, respectively. This enables the generation of high-quality images in various styles with minimal inference steps, without any further training.
• Figure 3: Generated images using LoRA and LCM-LoRA parameters applied to various painting styles. Starting with SDXL as the base model at 1024×1024 resolution, LoRA parameters (SD v1.5, SDXL, and SSD 1B) are fine-tuned on specific style datasets and combined with LCM-LoRA parameters. Image quality is assessed across multiple sampling steps. LoRA parameters utilize DPM-Solver++ sampler with $\omega = 8$ for classifier-free guidance, while combined parameters use LCM's multi-step sampler. The combination employs $\lambda_1 = 0.8$ and $\lambda_2 = 1.0$.