Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIBEVOICE-ASR Technical Report

Zhiliang Peng, Jianwei Yu, Yaoyao Chang, Zilong Wang, Li Dong, Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun, Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu Wei

TL;DR

VibeVoice-ASR tackles long-form speech understanding by addressing context fragmentation and multi-speaker complexity with a single-pass architecture that processes up to $60$ minutes of audio in one go. It unifies ASR, speaker diarization, and timestamping into an end-to-end Rich Transcription stream interleaving Who, When, and What, enabled by a $7.5$ Hz ultra-low frame-rate tokenizer feeding a decoder-only LLM. The approach incorporates a prompt-based context mechanism and extensive pre-training and supervised fine-tuning, including synthetic domain data and long-form transcription restoration, to achieve robust multilingual and code-switching performance. Experiments show state-of-the-art DER and tcpWER across multiple benchmarks, highlighting practical impact for long-form transcription tasks, with planned open-sourcing to foster community progress; limitations include multilingual forgetting and overlapping-speech handling.

Abstract

This report presents VibeVoice-ASR, a general-purpose speech understanding framework built upon VibeVoice, designed to address the persistent challenges of context fragmentation and multi-speaker complexity in long-form audio (e.g., meetings, podcasts) that remain despite recent advancements in short-form speech recognition. Unlike traditional pipelined approaches that rely on audio chunking, VibeVoice-ASRsupports single-pass processing for up to 60 minutes of audio. It unifies Automatic Speech Recognition, Speaker Diarization, and Timestamping into a single end-to-end generation task. In addition, VibeVoice-ASR supports over 50 languages, requires no explicit language setting, and natively handles code-switching within and across utterances. Furthermore, we introduce a prompt-based context injection mechanism that allows users to supply customized conetxt, significantly improving accuracy on domain-specific terminology and polyphonic character disambiguation.

VIBEVOICE-ASR Technical Report

TL;DR

VibeVoice-ASR tackles long-form speech understanding by addressing context fragmentation and multi-speaker complexity with a single-pass architecture that processes up to minutes of audio in one go. It unifies ASR, speaker diarization, and timestamping into an end-to-end Rich Transcription stream interleaving Who, When, and What, enabled by a Hz ultra-low frame-rate tokenizer feeding a decoder-only LLM. The approach incorporates a prompt-based context mechanism and extensive pre-training and supervised fine-tuning, including synthetic domain data and long-form transcription restoration, to achieve robust multilingual and code-switching performance. Experiments show state-of-the-art DER and tcpWER across multiple benchmarks, highlighting practical impact for long-form transcription tasks, with planned open-sourcing to foster community progress; limitations include multilingual forgetting and overlapping-speech handling.

Abstract

This report presents VibeVoice-ASR, a general-purpose speech understanding framework built upon VibeVoice, designed to address the persistent challenges of context fragmentation and multi-speaker complexity in long-form audio (e.g., meetings, podcasts) that remain despite recent advancements in short-form speech recognition. Unlike traditional pipelined approaches that rely on audio chunking, VibeVoice-ASRsupports single-pass processing for up to 60 minutes of audio. It unifies Automatic Speech Recognition, Speaker Diarization, and Timestamping into a single end-to-end generation task. In addition, VibeVoice-ASR supports over 50 languages, requires no explicit language setting, and natively handles code-switching within and across utterances. Furthermore, we introduce a prompt-based context injection mechanism that allows users to supply customized conetxt, significantly improving accuracy on domain-specific terminology and polyphonic character disambiguation.
Paper Structure (13 sections, 1 equation, 3 figures, 2 tables)

This paper contains 13 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Speech Tokenizer
  5. VibeVoice-ASR
  6. Pre-training
  7. Supervised Fine-Tuning (SFT)
  8. High-Quality Speech and Music Benchmarks.
  9. Context-Aware Synthetic Data Pipeline.
  10. Long-Form Transcription Restoration.
  11. Results
  12. Conclusion and Limitations
  13. Language Distribution of Training Data

Figures (3)

  • Figure 1: VibeVoice-ASR sets a new state-of-the-art for long-form speech understanding, consistently outperforming strong closed-source multimodal models (Gemini-2.5/3-Pro) across five public benchmarks. The results demonstrate superior accuracy in both speaker attribution (DER) and time-aligned transcription (tcpWER), particularly in complex multi-speaker environments.
  • Figure 2: The architectural overview of VibeVoice-ASR. VibeVoice-ASR processes 60-minute long-form audio in a single pass by ingesting continuous latents from dual-tokenizers alongside optional user-provided context. The output is a generated stream of Rich Transcription, explicitly interleaving Speaker ID (Who), Timestamps (When), and Content (What)
  • Figure 3: Language distribution in the training data.