Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models

Wenming Tu, Guanrou Yang, Ruiqi Yan, Wenxi Chen, Ziyang Ma, Yipeng Kang, Kai Yu, Xie Chen, Zilong Zheng

TL;DR

UltraVoice introduces the first large-scale, multi-dimensional speech style control dataset for end-to-end spoken dialogue. By fine-tuning mainstream models on UltraVoice, the work demonstrates substantial gains in instruction-following for style control and improvements to core conversational abilities, as validated on the URO-Bench benchmark and controllable TTS tasks. The dataset spans six stylistic dimensions and over 830 hours, and its construction combines a four-step pipeline with strict quality filtering. The results highlight a practical path toward more natural, expressive dialogue systems and broaden the applicability of controllable synthesis across speech tasks.

Abstract

Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like reasoning and question answering. To address this limitation, we introduce UltraVoice, the first large-scale speech dialogue dataset engineered for multiple fine-grained speech style control. Encompassing over 830 hours of speech dialogues, UltraVoice provides instructions across six key speech stylistic dimensions: emotion, speed, volume, accent, language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench benchmark, our fine-tuned models demonstrate substantial gains in core understanding, reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the dataset's utility extends to training controllable Text-to-Speech (TTS) models, underscoring its high quality and broad applicability for expressive speech synthesis. The complete dataset and model checkpoints are available at: https://github.com/bigai-nlco/UltraVoice.

UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models

TL;DR

UltraVoice introduces the first large-scale, multi-dimensional speech style control dataset for end-to-end spoken dialogue. By fine-tuning mainstream models on UltraVoice, the work demonstrates substantial gains in instruction-following for style control and improvements to core conversational abilities, as validated on the URO-Bench benchmark and controllable TTS tasks. The dataset spans six stylistic dimensions and over 830 hours, and its construction combines a four-step pipeline with strict quality filtering. The results highlight a practical path toward more natural, expressive dialogue systems and broaden the applicability of controllable synthesis across speech tasks.

Abstract

Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like reasoning and question answering. To address this limitation, we introduce UltraVoice, the first large-scale speech dialogue dataset engineered for multiple fine-grained speech style control. Encompassing over 830 hours of speech dialogues, UltraVoice provides instructions across six key speech stylistic dimensions: emotion, speed, volume, accent, language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench benchmark, our fine-tuned models demonstrate substantial gains in core understanding, reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the dataset's utility extends to training controllable Text-to-Speech (TTS) models, underscoring its high quality and broad applicability for expressive speech synthesis. The complete dataset and model checkpoints are available at: https://github.com/bigai-nlco/UltraVoice.
Paper Structure (29 sections, 4 figures, 14 tables)

This paper contains 29 sections, 4 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. End-to-End Spoken Dialogue Models
  4. Spoken Dialogue and Controllable TTS Dataset
  5. The UltraVoice Dataset
  6. Characteristics and Statistics
  7. Quality Assessment
  8. Experiment
  9. Experiment Setup
  10. Performance on Fine-grained Speech Style Control
  11. General Conversational Ability
  12. Validating Data Quality via Controllable Text-To-Speech
  13. Conclusion
  14. Statement For The Use of Large Language Models (LLMs)
  15. Detailed Dataset Statistics
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Overview of the UltraVoice Dataset Construction and Stylistic Coverage. The upper left section details the four-step process: text corpus curation, style injection & response generation, stylized speech synthesis, and quality control & filtering. The ring chart on the right visualizes the dataset’s control dimensions (inner ring) and their finer control sub-dimensions (outer ring). The lower panel provides examples of six speech style dimensions, including emotion, speed, volume, language, accent, and composite styles (e.g., combinations of speed, volume, and emotion).
  • Figure 2: Distributions of Duration and Number of Words.
  • Figure 3: Statistical visualizations of the six fine-grained speech style control dimensions in UltraVoice. The visualization methods are tailored to the nature of each dimension: t-SNE plots for categorical attributes (Emotion, Accent) demonstrate clear class separability; distributions of physical metrics (Speed, Volume) confirm precise control over acoustic properties; and word clouds (Language, Composite) highlight lexical diversity and expressive richness.
  • Figure 4: IFR (%) results across six fine-grained speech style control dimensions for each model. Each radar chart contrasts the base model (Blue) and its SFT variant (Red), with GPT-4o (Gray) used as an upper-bound reference.