Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models

Jae-Won Chung, Jeff J. Ma, Jisang Ahn, Yizhuo Liang, Akshay Jajoo, Myungjin Lee, Mosharaf Chowdhury

Abstract

Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81$\times$ higher throughput and 5.79$\times$ lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.

Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models

Abstract

Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81 higher throughput and 5.79 lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.
Paper Structure (19 sections, 5 figures, 1 table)

This paper contains 19 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. System Design
  3. Architecture Overview
  4. Control Plane.
  5. Data Plane.
  6. Deployment Lifecycle.
  7. Request Lifecycle.
  8. Task Abstraction and Model Fission
  9. Tasks and Apps.
  10. Model Fission.
  11. Record-and-Replay Execution.
  12. Component Sharing.
  13. Data Forwarding.
  14. Evaluation
  15. Setup.
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Computation graphs of (a) InternVL 3 internvl3-arxiv25, a multimodal input model, and (b) Qwen Omni qwen2.5-omni-arxiv25qwen3-omni-arxiv25, a multimodal input and output model. Different requests invoke different components and take different paths on the graph. $E$ stands for Encoder, $L$ for LLM, and $G$ for Generator. $L_{\text{th}}$ and $L_{\text{ta}}$ stand for thinker and talker LLMs, respectively.
  • Figure 2: Cornserve architecture and flows for deployment (black) and inference (red).
  • Figure 3: Monolith vs. Cornserve comparisons for Qwen 2.5 Omni 7B qwen2.5-omni-arxiv25 throughput and latency CDF, and Qwen 3 Omni 30B qwen3-omni-arxiv25 throughput. $\times$ indicate GPU out-of-memory.
  • Figure 4: Cornserve deployment configurations for Qwen 3 Omni on 8 and 16 GPUs. Each box represents a GPU. Model fission allows each component to scale independently: the thinker (LLM) uses tensor parallelism while talkers and generators are replicated to balance throughput.
  • Figure :