Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fun-Audio-Chat Technical Report

Tongyi Fun Team, Qian Chen, Luyao Cheng, Chong Deng, Xiangang Li, Jiaqing Liu, Chao-Hong Tan, Wen Wang, Junhao Xu, Jieping Ye, Qinglin Zhang, Qiquan Zhang, Jingren Zhou

TL;DR

Fun-Audio-Chat advances joint speech-text modeling by deploying Dual-Resolution Speech Representations that process audio at 5Hz for the backbone while generating 25Hz speech tokens, achieving substantial efficiency without compromising quality. A Core-Cocktail Training regime mitigates catastrophic forgetting through staged fine-tuning and model merging, followed by Multi-Task DPO to enhance robustness, instruction-following, and voice empathy. Built on pre-trained components with a multi-stage post-training pipeline, it demonstrates competitive to superior performance across Spoken QA, Audio Understanding, Speech Function Calling, and Voice Empathy, at 8B and 30B scales, and extends to a strong full-duplex variant. The work provides open-source releases of Fun-Audio-Chat-8B and an interactive demo, highlighting practical impact for scalable, capable voice-enabled AI systems.

Abstract

Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo.

Fun-Audio-Chat Technical Report

TL;DR

Fun-Audio-Chat advances joint speech-text modeling by deploying Dual-Resolution Speech Representations that process audio at 5Hz for the backbone while generating 25Hz speech tokens, achieving substantial efficiency without compromising quality. A Core-Cocktail Training regime mitigates catastrophic forgetting through staged fine-tuning and model merging, followed by Multi-Task DPO to enhance robustness, instruction-following, and voice empathy. Built on pre-trained components with a multi-stage post-training pipeline, it demonstrates competitive to superior performance across Spoken QA, Audio Understanding, Speech Function Calling, and Voice Empathy, at 8B and 30B scales, and extends to a strong full-duplex variant. The work provides open-source releases of Fun-Audio-Chat-8B and an interactive demo, highlighting practical impact for scalable, capable voice-enabled AI systems.

Abstract

Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo.
Paper Structure (28 sections, 7 equations, 6 figures, 7 tables)

This paper contains 28 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Speech Tokenization and Detokenization
  4. Dual-Resolution Speech Representations (DRSR)
  5. Multimodal Large Language Model (MLLM)
  6. Post-Training
  7. Pre-Alignment.
  8. Core-Cocktail Training.
  9. Multi-Task DPO Training.
  10. Full-duplex Interaction Training.
  11. Experiments
  12. Experimental Setup
  13. Evaluation Datasets.
  14. Evaluation Metrics.
  15. Baselines.
  16. ...and 13 more sections

Figures (6)

  • Figure : (a) Performance comparison on Spoken QA tasks.
  • Figure : (a) Fun-Audio-Chat architecture.
  • Figure : (a) Performance comparison on Spoken QA tasks.
  • Figure : (b) Performance comparison on other tasks.
  • Figure : (a) Fun-Audio-Chat architecture.
  • ...and 1 more figures