Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing

Lingxiao Zhao, Haoran Zhou, Yuezhi Che, Dazhao Cheng

TL;DR

<3-5 sentence high-level summary>: The paper addresses the end-to-end latency and throughput bottlenecks in multimodal LLM (MLLM) serving, caused by heavy video decoding and inter-stage blocking from vision encoding. It introduces two complementary designs: FlashCodec for collaborative multi-GPU video decoding to drastically reduce TTFT, and UnifiedServe for logically decoupled yet physically shared GPU scheduling to eliminate inter-stage blocking and maximize utilization. Together, they form an end-to-end optimized stack that achieves up to 3.0x more requests, up to 1.5x tighter SLOs, and up to 4.4x higher throughput relative to state-of-the-art baselines. The approach demonstrates substantial improvements in decoding capacity, SLO compliance, and end-to-end latency, enabling more scalable and responsive MLLM service deployments.

Abstract

Multimodal large language models (MLLMs) extend LLMs with visual understanding through a three-stage pipeline: multimodal preprocessing, vision encoding, and LLM inference. While these stages enhance capability, they introduce significant system bottlenecks. First, multimodal preprocessing-especially video decoding-often dominates Time-to-First-Token (TTFT). Most systems rely on CPU-based decoding, which severely limits throughput, while existing GPU-based approaches prioritize throughput-oriented parallelism and fail to meet the latency-sensitive requirements of MLLM inference. Second, the vision encoder is a standalone, compute-intensive stage that produces visual embeddings and cannot be co-batched with LLM prefill or decoding. This heterogeneity forces inter-stage blocking and increases token-generation latency. Even when deployed on separate GPUs, these stages underutilize available compute and memory resources, reducing overall utilization and constraining system throughput. To address these challenges, we present FlashCodec and UnifiedServe, two complementary designs that jointly optimize the end-to-end MLLM pipeline. FlashCodec accelerates the multimodal preprocessing stage through collaborative multi-GPU video decoding, reducing decoding latency while preserving high throughput. UnifiedServe optimizes the vision-to-text and inference stages using a logically decoupled their execution to eliminate inter-stage blocking, yet physically sharing GPU resources to maximize GPU system utilization. By carefully orchestrating execution across stages and minimizing interference, UnifiedServe Together, our proposed framework forms an end-to-end optimized stack that can serve up to 3.0$\times$ more requests or enforce 1.5$\times$ tighter SLOs, while achieving up to 4.4$\times$ higher throughput compared to state-of-the-art systems.

Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing

TL;DR

<3-5 sentence high-level summary>: The paper addresses the end-to-end latency and throughput bottlenecks in multimodal LLM (MLLM) serving, caused by heavy video decoding and inter-stage blocking from vision encoding. It introduces two complementary designs: FlashCodec for collaborative multi-GPU video decoding to drastically reduce TTFT, and UnifiedServe for logically decoupled yet physically shared GPU scheduling to eliminate inter-stage blocking and maximize utilization. Together, they form an end-to-end optimized stack that achieves up to 3.0x more requests, up to 1.5x tighter SLOs, and up to 4.4x higher throughput relative to state-of-the-art baselines. The approach demonstrates substantial improvements in decoding capacity, SLO compliance, and end-to-end latency, enabling more scalable and responsive MLLM service deployments.

Abstract

Multimodal large language models (MLLMs) extend LLMs with visual understanding through a three-stage pipeline: multimodal preprocessing, vision encoding, and LLM inference. While these stages enhance capability, they introduce significant system bottlenecks. First, multimodal preprocessing-especially video decoding-often dominates Time-to-First-Token (TTFT). Most systems rely on CPU-based decoding, which severely limits throughput, while existing GPU-based approaches prioritize throughput-oriented parallelism and fail to meet the latency-sensitive requirements of MLLM inference. Second, the vision encoder is a standalone, compute-intensive stage that produces visual embeddings and cannot be co-batched with LLM prefill or decoding. This heterogeneity forces inter-stage blocking and increases token-generation latency. Even when deployed on separate GPUs, these stages underutilize available compute and memory resources, reducing overall utilization and constraining system throughput. To address these challenges, we present FlashCodec and UnifiedServe, two complementary designs that jointly optimize the end-to-end MLLM pipeline. FlashCodec accelerates the multimodal preprocessing stage through collaborative multi-GPU video decoding, reducing decoding latency while preserving high throughput. UnifiedServe optimizes the vision-to-text and inference stages using a logically decoupled their execution to eliminate inter-stage blocking, yet physically sharing GPU resources to maximize GPU system utilization. By carefully orchestrating execution across stages and minimizing interference, UnifiedServe Together, our proposed framework forms an end-to-end optimized stack that can serve up to 3.0 more requests or enforce 1.5 tighter SLOs, while achieving up to 4.4 higher throughput compared to state-of-the-art systems.

Paper Structure

This paper contains 31 sections, 18 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. MLLM Inference Overview
  4. Toward Low-Latency, High-Throughput Video Decoding for MLLM Serving
  5. Limitations of Current CPU and GPU Decoding.
  6. Collaborative GPU Decoding is the Key.
  7. Overcoming Asymmetric Interference through Multi-GPU Scaling.
  8. Toward Unified and Interference-Aware MLLM Scheduling
  9. Scheduling Policies for LLM Inference.
  10. Inefficiency of Current Scheduling on MLLMs.
  11. Our Idea: Logically Decoupling the Pipeline Stages while Physically Sharing GPU Resources.
  12. Challenge: Characterizing Resource Contention
  13. FlashCodec Techniques
  14. Introduction to Video Decoding.
  15. GOP-bsed Parallel Decoding
  16. ...and 16 more sections

Figures (18)

  • Figure 1: Qwen2.5-VL-32B running on four A100 GPUs serving 30 video requests from MLVU dataset. (a) Both systems experienced frequent generation stalls. (b) The overall performance across all systems
  • Figure 2: MLLM inference process overview.
  • Figure 3: Performance comparison across different configurations, with Decord decord_decoder as the default CPU/GPU decoder.
  • Figure 4: (a) Normalized performance of video decoding and LLM serving under different parallel configurations (PC). (b) Video decoding latency value.
  • Figure 5: Comparison of three scheduling pipelines.
  • ...and 13 more figures