Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Siqi Zheng

TL;DR

FunAudioLLM presents SenseVoice and CosyVoice as a complementary pair of foundation models to enable natural voice interactions with LLMs. SenseVoice delivers rapid multilingual understanding (ASR, LID, SER, AED) across languages, while CosyVoice provides high-quality multilingual speech generation with zero-shot learning, cross-lingual cloning, and instruction-based controllability via the S^3 tokenizer. The paper demonstrates strong multilingual recognition, emotion handling, and audio-event detection, along with competitive or human-parity generation quality and the ability to augment data for ASR. Open-source releases, training/inference code, and demos broaden the practical impact for speech-enabled dialogue systems, interactive media, and cross-language communication.

Abstract

This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

TL;DR

FunAudioLLM presents SenseVoice and CosyVoice as a complementary pair of foundation models to enable natural voice interactions with LLMs. SenseVoice delivers rapid multilingual understanding (ASR, LID, SER, AED) across languages, while CosyVoice provides high-quality multilingual speech generation with zero-shot learning, cross-lingual cloning, and instruction-based controllability via the S^3 tokenizer. The paper demonstrates strong multilingual recognition, emotion handling, and audio-event detection, along with competitive or human-parity generation quality and the ability to augment data for ASR. Open-source releases, training/inference code, and demos broaden the practical impact for speech-enabled dialogue systems, interactive media, and cross-language communication.

Abstract

This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.
Paper Structure (26 sections, 2 equations, 13 figures, 14 tables)

This paper contains 26 sections, 2 equations, 13 figures, 14 tables.

Table of Contents

  1. Introduction
  2. FunAudioLLM Models
  3. Overview of FunAudioLLM
  4. Voice Understanding Model: SenseVoice
  5. Semantic Speech Tokenizer
  6. Voice Generation Model: CosyVoice
  7. System Overview
  8. Model Training
  9. Zero-shot In-context Learning
  10. Instruction Fine-tuning
  11. Dataset
  12. Training Set for SenseVoice
  13. Training Set for CosyVoice
  14. Experimental Results
  15. Multilingual Speech Recognition
  16. ...and 11 more sections

Figures (13)

  • Figure 1: An overview of our FunAudioLLM models for voice understanding and generation.
  • Figure 2: SenseVoice is a comprehensive speech foundation model designed to perform various speech understanding tasks, including Automatic Speech Recognition (ASR), Language Identification (LID), Speech Emotion Recognition (SER), and Audio Event Detection (AED). SenseVoice-Small [Top]: An encoder-only model optimized for rapid speech understanding. It offers high-speed processing while supporting 5 languages. SenseVoice-Large [Bottom]: An encoder-decoder model aimed at achieving more precise speech understanding across a broader range of languages. It excels in accuracy and supports an extensive set of language capabilities.
  • Figure 3: An illustration of our supervised semantic speech tokenizer.
  • Figure 4: A semantic diagram of CosyVoice models.
  • Figure 5: Sequence construction for (a) zero-shot in-context learning and (b) cross-lingual voice cloning. LID represents language identifier.
  • ...and 8 more figures