Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data

Shuaijiang Zhao, Tingwei Guo, Bajian Xiang, Tongtang Wan, Qiang Niu, Wei Zou, Xiangang Li

TL;DR

KE-Omni introduces a seamless, real-time bilingual speech language model built on the 7-million-dialogue Ke-SpeechChat dataset, featuring over 60,000 hours and 42,002 speakers. The architecture combines a Whisper-based speech encoder, a LLaMA-3.1-8B-Instruct LLM, and a HuBERT-derived, chunked speech decoder with a HiFi-GAN vocoder to enable low-latency, natural-sounding speech responses in Chinese and English. The authors detail a scalable pipeline for synthetic dialogue creation (text rewriting, filtering, and spoken-style processing), plus a diverse virtual voice library and rigorous quality control, to address data scarcity and privacy concerns. Across S2TIF, modality alignment, and speech quality benchmarks, KE-Omni outperforms baselines and demonstrates the value of large-scale synthetic speech interaction data for advancing real-time speech-language modeling; code and models are planned for release after risk assessment.

Abstract

The GPT-4o represents a significant milestone in enabling real-time interaction with large language models (LLMs) through speech, its remarkable low latency and high fluency not only capture attention but also stimulate research interest in the field. This real-time speech interaction is particularly valuable in scenarios requiring rapid feedback and immediate responses, dramatically enhancing user experience. However, there is a notable lack of research focused on real-time large speech language models, particularly for Chinese. In this work, we present KE-Omni, a seamless large speech language model built upon Ke-SpeechChat, a large-scale high-quality synthetic speech interaction dataset consisting of 7 million Chinese and English conversations, featuring 42,002 speakers, and totaling over 60,000 hours, This contributes significantly to the advancement of research and development in this field. The demos can be accessed at \url{https://huggingface.co/spaces/KE-Team/KE-Omni}.

Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data

TL;DR

KE-Omni introduces a seamless, real-time bilingual speech language model built on the 7-million-dialogue Ke-SpeechChat dataset, featuring over 60,000 hours and 42,002 speakers. The architecture combines a Whisper-based speech encoder, a LLaMA-3.1-8B-Instruct LLM, and a HuBERT-derived, chunked speech decoder with a HiFi-GAN vocoder to enable low-latency, natural-sounding speech responses in Chinese and English. The authors detail a scalable pipeline for synthetic dialogue creation (text rewriting, filtering, and spoken-style processing), plus a diverse virtual voice library and rigorous quality control, to address data scarcity and privacy concerns. Across S2TIF, modality alignment, and speech quality benchmarks, KE-Omni outperforms baselines and demonstrates the value of large-scale synthetic speech interaction data for advancing real-time speech-language modeling; code and models are planned for release after risk assessment.

Abstract

The GPT-4o represents a significant milestone in enabling real-time interaction with large language models (LLMs) through speech, its remarkable low latency and high fluency not only capture attention but also stimulate research interest in the field. This real-time speech interaction is particularly valuable in scenarios requiring rapid feedback and immediate responses, dramatically enhancing user experience. However, there is a notable lack of research focused on real-time large speech language models, particularly for Chinese. In this work, we present KE-Omni, a seamless large speech language model built upon Ke-SpeechChat, a large-scale high-quality synthetic speech interaction dataset consisting of 7 million Chinese and English conversations, featuring 42,002 speakers, and totaling over 60,000 hours, This contributes significantly to the advancement of research and development in this field. The demos can be accessed at \url{https://huggingface.co/spaces/KE-Team/KE-Omni}.

Paper Structure

This paper contains 34 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Ke-SpeechChat Dataset
  4. Textual Dialogue Data
  5. Rewriting Instructions
  6. Filtering of Rewritten Instructions
  7. Spoken Style Post-Processing
  8. Speech Dialogue Data
  9. Voice Library
  10. Speech Synthesis
  11. Quality Assurance
  12. Details of Ke-SpeechChat
  13. Metadata
  14. Statistics
  15. Partitions
  16. ...and 19 more sections

Figures (4)

  • Figure 1: An overview of textual dialogue data process
  • Figure 2: The construction of speech dialogue data. (A) Construction of real speakers. (B) Construction of virtual speakers. (C) Speech generation given textual dialogues and virtual speakers. (D) Quality filtering and assessment.
  • Figure 3: Model architecture of KE-Omni.
  • Figure 4: Evaluation methodology for large speech language model, focusing on three key capabilities: Speech-to-Text Instruction-Following(S2TIF), Modality Alignment and Speech Quality.