EdgeFusion: On-Device Text-to-Image Generation

TL;DR

EdgeFusion tackles the practical barrier of deploying diffusion-based T2I on edge hardware by combining a compact BK-SDM backbone with an advanced LCM distillation workflow and high-quality, AI-generated training data. The approach includes a teacher-guided BK-SDM-Adv-Tiny initialization, staged data enhancement (from LAION preprocessing to fully synthetic and curated datasets), and edge-centric deployment techniques such as model-level tiling and mixed-precision quantization. Results show significant gains in image-text alignment and image quality, with EdgeFusion achieving near-parity with the original tiny SD model while delivering two-step or four-step generation in under one second on Samsung Exynos NPUs, and substantial speedups on GPUs. The work demonstrates the importance of high-quality synthetic data and deployment-aware optimizations for practical, on-device text-to-image synthesis, enabling realistic, real-time T2I on resource-constrained devices.

Abstract

The intensive computational burden of Stable Diffusion (SD) for text-to-image generation poses a significant hurdle for its practical application. To tackle this challenge, recent research focuses on methods to reduce sampling steps, such as Latent Consistency Model (LCM), and on employing architectural optimizations, including pruning and knowledge distillation. Diverging from existing approaches, we uniquely start with a compact SD variant, BK-SDM. We observe that directly applying LCM to BK-SDM with commonly used crawled datasets yields unsatisfactory results. It leads us to develop two strategies: (1) leveraging high-quality image-text pairs from leading generative models and (2) designing an advanced distillation process tailored for LCM. Through our thorough exploration of quantization, profiling, and on-device deployment, we achieve rapid generation of photo-realistic, text-aligned images in just two steps, with latency under one second on resource-limited edge devices.

Figures (7)

• Figure 1: T2I generation results. When trained with improved data, our EdgeFusion can produce high-quality images from challenging prompts in just a few denoising steps.
• Figure 2: A compact SD with step reduction. (a) Vanilla application of LCM: we initialize BK-LCM-Tiny with the weight from BK-SDM-Tiny and train with distillation to reduce sampling steps. (b) Our approach: improving the initialization of the LCM's student with a better teacher is beneficial. Moreover, in the LCM training phase, employing the original teacher enhances performance. Leveraging high-quality data is crucial in both phases.
• Figure 3: Comparison between FP32 and W8A16 quantized EdgeFusion (2 steps). Prompts can be found in Supplementary.
• Figure 4: Output examples from BK-SDM-Adv-Tiny. Prompts (left to right, top to bottom): "a cute cat, hyper detailed, cinematic light, ultra high res, best shadow, RAW", "a teenager in a black graphic tee, the design a splash of neon colors", "photo portrait of Will Smith, hyper detailed, ultra high res, RAW", "photo portrait of an old man with blond hair", "painting of fruits on a table, next to a bottle of wine and a glass", "a beautiful pizza", "A sleek black sedan parked in a garage.", "a fox, photo-realistic, high-resolution, 4k", "photo portrait of a man wearing a charcoal gray suit, crisp and meticulously tailored", "photo portrait of a young woman with blond hair", "flowers in a golden vase", "a lion is reading a book on the desk, photo-realistic", "an astronaut in the jungle", "A classic convertible parked on a sunny beach.", "photo portrait of Brad Pitt wearing a black hat".
• Figure 5: Output examples from EdgeFusion with 1 step inference. Prompts (left to right, top to bottom): "a bear wearing a hat and sunglasses, photo-realistic, high-resolution, 4k", "photo portrait of an old man with blond hair", "flowers in a golden vase", "a cute rabbit", "a frozen tundra under the aurora borealis, the night sky alive with color, ice crystals sparkling like diamonds", "close up on colorful flowers", "a beautiful sports car", "a mountain peak during golden hour", "photo portrait of an old woman dressed in red", "photo portrait of a young man with blond hair", "beautiful photo portrait of the king, high resolution, 4k", "a cute dog wearing a hat", "photo portrait of an old man dressed in white", "a cute cat wearing sunglasses, photo-realistic", "photo portrait of an old woman with blond hair".