Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sink or SWIM: Tackling Real-Time ASR at Scale

Federico Bruzzone, Walter Cazzola, Matteo Brancaleoni, Dario Pellegrino

TL;DR

SWIM tackles real-time, multi-client ASR scalability by building on OpenAI's Whisper and introducing true model-level parallelization through a shared audio buffer. It combines a buffer merging mechanism with a QRatio-based local agreement in a hypothesis buffer to maintain transcription fidelity while enabling parallel processing on a single Whisper instance. Empirical results show that SWIM preserves accuracy (WER and semantic similarity) while reducing latency, achieving roughly $2.4$ seconds delay with five concurrent clients and scalable performance up to $20$ concurrent streams across English, Italian, and Spanish. The work demonstrates a practical, cost-effective path for enterprise-grade, multilingual real-time transcription in dynamic, multi-user environments.

Abstract

Real-time automatic speech recognition systems are increasingly integrated into interactive applications, from voice assistants to live transcription services. However, scaling these systems to support multiple concurrent clients while maintaining low latency and high accuracy remains a major challenge. In this work, we present SWIM, a novel real-time ASR system built on top of OpenAI's Whisper model that enables true model-level parallelization for scalable, multilingual transcription. SWIM supports multiple concurrent audio streams without modifying the underlying model. It introduces a buffer merging strategy that maintains transcription fidelity while ensuring efficient resource usage. We evaluate SWIM in multi-client settings -- scaling up to 20 concurrent users -- and show that it delivers accurate real-time transcriptions in English, Italian, and Spanish, while maintaining low latency and high throughput. While Whisper-Streaming achieves a word error rate of approximately 8.2% with an average delay of approximately 3.4 s in a single-client, English-only setting, SWIM extends this capability to multilingual, multi-client environments. It maintains comparable accuracy with significantly lower delay -- around 2.4 s with 5 clients -- and continues to scale effectively up to 20 concurrent clients without degrading transcription quality and increasing overall throughput. Our approach advances scalable ASR by improving robustness and efficiency in dynamic, multi-user environments.

Sink or SWIM: Tackling Real-Time ASR at Scale

TL;DR

SWIM tackles real-time, multi-client ASR scalability by building on OpenAI's Whisper and introducing true model-level parallelization through a shared audio buffer. It combines a buffer merging mechanism with a QRatio-based local agreement in a hypothesis buffer to maintain transcription fidelity while enabling parallel processing on a single Whisper instance. Empirical results show that SWIM preserves accuracy (WER and semantic similarity) while reducing latency, achieving roughly seconds delay with five concurrent clients and scalable performance up to concurrent streams across English, Italian, and Spanish. The work demonstrates a practical, cost-effective path for enterprise-grade, multilingual real-time transcription in dynamic, multi-user environments.

Abstract

Real-time automatic speech recognition systems are increasingly integrated into interactive applications, from voice assistants to live transcription services. However, scaling these systems to support multiple concurrent clients while maintaining low latency and high accuracy remains a major challenge. In this work, we present SWIM, a novel real-time ASR system built on top of OpenAI's Whisper model that enables true model-level parallelization for scalable, multilingual transcription. SWIM supports multiple concurrent audio streams without modifying the underlying model. It introduces a buffer merging strategy that maintains transcription fidelity while ensuring efficient resource usage. We evaluate SWIM in multi-client settings -- scaling up to 20 concurrent users -- and show that it delivers accurate real-time transcriptions in English, Italian, and Spanish, while maintaining low latency and high throughput. While Whisper-Streaming achieves a word error rate of approximately 8.2% with an average delay of approximately 3.4 s in a single-client, English-only setting, SWIM extends this capability to multilingual, multi-client environments. It maintains comparable accuracy with significantly lower delay -- around 2.4 s with 5 clients -- and continues to scale effectively up to 20 concurrent clients without degrading transcription quality and increasing overall throughput. Our approach advances scalable ASR by improving robustness and efficiency in dynamic, multi-user environments.
Paper Structure (20 sections, 4 equations, 7 figures)

This paper contains 20 sections, 4 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. State-of-the-Art
  4. A Deep Dive into SWIM
  5. Overview
  6. Clients
  7. Services
  8. Shared ASR Model
  9. Hypothesis Buffer with Local Agreement
  10. Evaluation
  11. Datasets
  12. Experimental Setup
  13. Experiments and Results
  14. Concluding Remarks
  15. Threats to Validity
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Architecture of SWIM. Multiple clients ❶ stream audio to dedicated services ❷, which forward data to a shared ASR model ❸ for parallel processing.
  • Figure 2: Audio chunk processing time diagram.
  • Figure 3: Evaluation results with different chunk durations using box plots.
  • Figure 4: The similarity score distribution per client across all datasets using boxen plots.
  • Figure 5: Delay per language across the Multilingual LibriSpeech dataset and Google Fleurs dataset using violin plots.
  • ...and 2 more figures