Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Rodrigo Fonseca, Ricardo Bianchini

TL;DR

An adaptive, modular serving system that dynamically manages quality, model/content parallelism, model/content parallelism, and resource-aware scheduling, StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45, and quantifies the trade-offs between latency, cost, and quality.

Abstract

Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image generation from a prompt) in batch mode, often requiring several seconds even for basic results. Serving real-time multi-modal workflows at scale is costly and complex, requiring efficient coordination of diverse models (each with unique resource needs) across language, audio, image, and video, all under strict latency and resource constraints. We tackle these challenges through the lens of real-time podcast video generation, integrating LLMs, text-to-speech, and video-audio generation. To meet tight SLOs, we design an adaptive, modular serving system, StreamWise, that dynamically manages quality (e.g., resolution, sharpness), model/content parallelism, and resource-aware scheduling. We leverage heterogeneous hardware to maximize responsiveness and efficiency. For example, the system can lower video resolution and allocate more resources to early scenes. We quantify the trade-offs between latency, cost, and quality. The cheapest setup generates a 10-minute podcast video on A100 GPUs in 1.4 hours (8.4x slower than the real-time) for less than \$25. StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45.

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

TL;DR

An adaptive, modular serving system that dynamically manages quality, model/content parallelism, model/content parallelism, and resource-aware scheduling, StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45, and quantifies the trade-offs between latency, cost, and quality.

Abstract

Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image generation from a prompt) in batch mode, often requiring several seconds even for basic results. Serving real-time multi-modal workflows at scale is costly and complex, requiring efficient coordination of diverse models (each with unique resource needs) across language, audio, image, and video, all under strict latency and resource constraints. We tackle these challenges through the lens of real-time podcast video generation, integrating LLMs, text-to-speech, and video-audio generation. To meet tight SLOs, we design an adaptive, modular serving system, StreamWise, that dynamically manages quality (e.g., resolution, sharpness), model/content parallelism, and resource-aware scheduling. We leverage heterogeneous hardware to maximize responsiveness and efficiency. For example, the system can lower video resolution and allocate more resources to early scenes. We quantify the trade-offs between latency, cost, and quality. The cheapest setup generates a 10-minute podcast video on A100 GPUs in 1.4 hours (8.4x slower than the real-time) for less than \45.
Paper Structure (25 sections, 1 equation, 16 figures, 4 tables)

This paper contains 25 sections, 1 equation, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Podcast video generation
  3. Workflow components
  4. Other workflows
  5. Goals for multi-modal workflow serving
  6. Monolithic integrated models vs. workflows
  7. Multi-modal models characterization
  8. Generative multi-modal models
  9. Performance breakdown
  10. Impact of hardware
  11. Distributed inference across multiple servers
  12. Serving multi-modal generation
  13. Systems opportunities for efficiency
  14. StreamWise overview
  15. On-boarding multi-modal models
  16. ...and 10 more sections

Figures (16)

  • Figure 1: Workflow for video podcast generation.
  • Figure 2: Video generation architecture using diffusion transformers and VAE with text, image, and audio inputs.
  • Figure 3: Latency to generate an 81-frame (5.1-second at 16 FPs) video at $640\times{}400$ (16:10 aspect ratio) using 10 diffusion steps on A100 GPUs. It includes the sensitivity to each of these parameters. The red dashed line indicates real time.
  • Figure 4: Latency sensitivity to the NVIDIA GPU generation with 4 GPUs in a single server (limited to match GB200).
  • Figure 5: Sensitivity to the number of H200 GPUs across servers using USP with Ulysses and Ring attention.
  • ...and 11 more figures