TetriServe: Efficient DiT Serving for Heterogeneous Image Generation

TL;DR

DiT serving under strict SLOs is challenging due to heterogeneous resolutions and shared GPUs. TetriServe addresses this with step-level sequence parallelism and a deadline-aware round-based scheduler, guided by an offline cost model and a DP-based packing strategy, to minimize GPU hours while maximizing the number of requests meeting deadlines. The approach yields up to 32% higher SLO attainment than fixed-parallelism baselines across state-of-the-art DiT models and hardware, and remains robust under bursty traffic and mixed workloads. This work provides a practical, scalable solution for production DiT serving with concrete performance gains and a clear optimization framework for deadline-driven resource allocation.

Abstract

Diffusion Transformer (DiT) models excel at generating highquality images through iterative denoising steps, but serving them under strict Service Level Objectives (SLOs) is challenging due to their high computational cost, particularly at large resolutions. Existing serving systems use fixed degree sequence parallelism, which is inefficient for heterogeneous workloads with mixed resolutions and deadlines, leading to poor GPU utilization and low SLO attainment. In this paper, we propose step-level sequence parallelism to dynamically adjust the parallel degree of individual requests according to their deadlines. We present TetriServe, a DiT serving system that implements this strategy for highly efficient image generation. Specifically, TetriServe introduces a novel round-based scheduling mechanism that improves SLO attainment: (1) discretizing time into fixed rounds to make deadline-aware scheduling tractable, (2) adapting parallelism at the step level and minimize GPU hour consumption, and (3) jointly packing requests to minimize late completions. Extensive evaluation on state-of-the-art DiT models shows that TetriServe achieves up to 32% higher SLO attainment compared to existing solutions without degrading image quality.

Figures (13)

• Figure 1: Three DiT serving requests---each with 5 denoising steps---arrive over time with different SLOs and output resolutions. DiT serving solutions using static parallelism cannot adapt and fail to meet multiple SLOs. TetriServe meets more SLOs via SLO-aware scheduling and packing.
• Figure 2: End-to-end scaling efficiency of FLUX.1-dev flux1dev2024 for four resolutions on an 8$\times$H100 server for different batch size (BS). Efficiency scales sublinearly. Larger resolutions benefit more from increased parallelism, while smaller resolutions exhibit limited scalability. Note different Y-axes scales.
• Figure 3: Performance of fixed degree xDiT variants under the Uniform workload. (a) The overall SLO Attainment Ratio (SAR) is low for all fixed strategies. (b) The spider plot, shown for a representative arrival rate of 12 req/min, reveals the underlying reason: low SP degrees fail on large resolutions, while high SP degrees perform poorly on small ones.
• Figure 4: System overview of TetriServe.
• Figure 5: Illustration of TetriServe's scheduling process. The progression is shown from top to bottom: each row represents an intermediate scheduling step, while the final row shows the actual GPU allocation decision. Time is fixed across rows.