DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling

TL;DR

DiffServe tackles the challenge of efficiently serving text-to-image diffusion models under fluctuating query demand by introducing query-aware model scaling through light–heavy diffusion cascades and a learned discriminator to route prompts by complexity. A MILP-based resource allocator optimizes the distribution of hardware, batch sizes, and the cascade’s confidence threshold, guided by latency and throughput constraints that account for queuing delays via Little's law. The discriminator, built on EfficientNet and trained with real images, enables accurate quality assessment to minimize unnecessary deferrals to heavier models, while the MILP balances quality and throughput. Across synthetic and real traces, DiffServe reports up to 24% improvements in image quality and up to 70% reductions in SLO violations relative to baselines, demonstrating scalable, production-ready diffusion model serving.

Abstract

Text-to-image generation using diffusion models has gained increasing popularity due to their ability to produce high-quality, realistic images based on text prompts. However, efficiently serving these models is challenging due to their computation-intensive nature and the variation in query demands. In this paper, we aim to address both problems simultaneously through query-aware model scaling. The core idea is to construct model cascades so that easy queries can be processed by more lightweight diffusion models without compromising image generation quality. Based on this concept, we develop an end-to-end text-to-image diffusion model serving system, DiffServe, which automatically constructs model cascades from available diffusion model variants and allocates resources dynamically in response to demand fluctuations. Our empirical evaluations demonstrate that DiffServe achieves up to 24% improvement in response quality while maintaining 19-70% lower latency violation rates compared to state-of-the-art model serving systems.

Figures (9)

• Figure 1: (a) The quality-latency trade-offs of systems serving independent diffusion models and diffusion model cascades with different discriminator designs with batch size one. The top panel uses diffusion model cascades built with SDv1.5 as the heavyweight model (H) and SD-Turbo as the lightweight model (L). The bottom panel uses SDXS as the lightweight model instead. Lower latency is achieved by using lighter models or treating more queries as easy in diffusion model cascades. FID is the lower the better. (b) The distribution of the difference in image quality between a lightweight model and a heavyweight model. Negative values in the x-axis mean the lightweight model's generated image quality is better than those from the heavyweight model. Top panels use PickScore as the quality metric while bottom panels use confidence score from our proposed discriminator. (c) Illustration of how different resource allocation configurations affect serving throughput (QPS) and response quality (FID). All results use the dataset MS-COCO 2017 lin2014microsoft.
• Figure 2: System architecture of DiffServe: (1) The query from the client is sent to the load balancer, (2) The load balancer sends it to a worker with the lightweight model and the discriminator, (3) If the confidence score is greater than the threshold, the response is sent back to client, (4) Else, the query is sent to a worker with the heavyweight model, and (5) its output is sent to the client.
• Figure 3: Training and inference paths of the discriminator. 'Real' refers to images sourced from real-world high-quality datasets. 'Fake' refers to the generated images from diffusion models. GLM: Generated images from Lightweight diffusion Model; GHM: Generated images from Heavyweight diffusion Model.
• Figure 4: Performance comparison on static trace. DiffServe offers Pareto optimal between FID and SLO violations (lower left curve).
• Figure 5: Performance comparison on real-world trace for Cascade 1. DiffServe improves quality by up to 23.4% over baselines while maintaining low SLO violations. During peak, it offers similar or better quality than static approaches with significantly lower SLO violations (from 19-70%).