EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend

TL;DR

EPD-Serve tackles inefficiencies in multimodal large-model inference by decoupling Encode, Prefill, and Decode into independently schedulable stages. It introduces asynchronous cross-stage transmission and hierarchical KV transfer, plus modality-aware multi-path scheduling and flexible physical co-location to optimize throughput while meeting strict SLOs ($TTFT \leq 2000\,ms$, $TPOT \leq 50\,ms$). Through detailed experiments on Ascend hardware with openPangu-7B-VL and Qwen3-VL-8B across multiple deployments, the work demonstrates substantial throughput gains (up to ~69% at high load) and robust tail-latency performance, validating stage-level disaggregation as a principled optimization. The findings offer practical deployment guidance, showing encode-focused disaggregation improves TTFT while decode-focused disaggregation stabilizes TPOT, and that full Encode-Prefill-Decode disaggregation achieves the best overall throughput under relaxed latency constraints. Overall, EPD-Serve provides a scalable framework for heterogeneous multimodal workloads by combining logical isolation with physical co-location and advanced transmission techniques.

Abstract

With the widespread adoption of large multimodal models, efficient inference across text, image, audio, and video modalities has become critical. However, existing multimodal inference systems typically employ monolithic architectures that tightly couple the Encode, Prefill, and Decode stages on homogeneous hardware, neglecting the heterogeneous computational characteristics of each stage. This design leads to inefficient resource utilization and limited system throughput. To address these issues, we propose EPD-Serve, a stage-level disaggregated inference serving system for multimodal models. EPD-Serve decouples the inference pipeline into independent Encode, Prefill, and Decode stages, enabling logical isolation and flexible co-located deployment through dynamic orchestration. Leveraging the Ascend interconnect topology, EPD-Serve introduces asynchronous feature prefetching between Encode and Prefill stages and a hierarchical grouped KV cache transmission mechanism between Prefill and Decode stages to improve cross-node communication efficiency. In addition, EPD-Serve incorporates multi-route scheduling, instance-level load balancing, and multi-stage hardware co-location with spatial multiplexing to better support diverse multimodal workloads. Comprehensive experiments on multimodal understanding models demonstrate that, under high-concurrency scenarios, EPD-Serve improves end-to-end throughput by 57.37-69.48% compared to PD-disaggregated deployment, while satisfying strict SLO constraints, including TTFT below 2000 ms and TPOT below 50 ms. These results highlight the effectiveness of stage-level disaggregation for optimizing multimodal large model inference systems.

Figures (17)

• Figure 1: Inference flow for large multi-modal language models.
• Figure 2: Latency proportion of mainstream MLLMs as encoder sequence length increases.
• Figure 3: Overview of the EPD-Serve disaggregated inference architecture.
• Figure 4: E-P disaggregation with asynchronous feature transmission and pipelining. Left: Asynchronous feature transmission between the Encode and Prefill stages via the Mooncake Store qin2025mooncake. Right: Pipeline layouts supporting standalone deployment or physical co-location with the Prefill stage.
• Figure 5: P-D disaggregation with hierarchical KV transmission and pipelining. Left: Hierarchically grouped KV transmission between the Prefill and Decode stages. Right: Pipelined overlap of KV transmission and computation to hide communication latency.