VoxServe: Streaming-Centric Serving System for Speech Language Models

TL;DR

VoxServe tackles the challenge of efficiently serving streaming Speech Language Models by introducing a unified model execution interface that decouples model architecture from system-level optimizations. It advances with streaming-aware scheduling and an asynchronous pipeline to optimize Time-To-First-Audio (TTFA) and uninterrupted playback (Streaming Viability) across diverse SpeechLMs. Empirical results across CosyVoice, Orpheus, and Step-Audio show 10–20x higher throughput at comparable latency, with robust scalability to multi-GPU deployments and other deployment scenarios. The work reduces engineering costs for supporting heterogeneous SpeechLMs and enables practical, high-quality streaming speech services at scale.

Abstract

Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.

Figures (11)

• Figure 1: Typical workflow of SpeechLM inference.
• Figure 2: SpeechLMs have diversity in how to represent both text and audio data, including number of codebooks, usage of continuous feature values from audio inputs, and the existence of depth-wise LLM.
• Figure 3: (Left) SpeechLM deployment is currently fragmented by bespoke, architecture-specific inference stacks, leading to suboptimal scheduling and resource management, and requires significant engineering cost to adopt a new architecture. (Right) We design a unified serving system that supports diverse SpeechLMs, which enables holistic system optimization.
• Figure 4: Overview of VoxServe architecture. The execution process has three modules: Scheduler for request orchestration, Worker for GPU management, and Model for providing a common abstraction across various SpeechLMs. Together, this design enables holistic and model-agnostic optimization of SpeechLM serving.
• Figure 5: Asynchronous pipeline design. VoxServe overlaps GPU computation with independent CPU-side tasks to reduce scheduling overhead.