Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hello-Chat: Towards Realistic Social Audio Interactions

Yueran Hou, Peilei Jia, Zihan Sun, Qihang Lu, Wenbing Yang, Yingming Gao, Ya Li, Jun Gao

TL;DR

Experimental results show that the Hello-Chat model not only reaches state-of-the-art (SOTA) performance on specific audio understanding tasks but also significantly outperforms existing baselines in prosodic naturalness and emotional alignment, paving the way for the next generation of empathetic AI agents.

Abstract

Recent advancements in Large Audio Language Models (LALMs) have demonstrated exceptional performance in speech recognition and translation. However, existing models often suffer from a disconnect between perception and expression, resulting in a robotic "read-speech" style that lacks the spontaneity and emotional resonance of real human interaction. In this report, we introduce Hello-Chat, an end-to-end audio language model designed for realistic social scenarios. By leveraging a massive dataset of real-life conversations and employing a modality-interleaved training strategy, Hello-Chat achieves a breakthrough in anthropomorphic generation. Experimental results show that our model not only reaches state-of-the-art (SOTA) performance on specific audio understanding tasks but also significantly outperforms existing baselines in prosodic naturalness and emotional alignment, paving the way for the next generation of empathetic AI agents.

Hello-Chat: Towards Realistic Social Audio Interactions

TL;DR

Experimental results show that the Hello-Chat model not only reaches state-of-the-art (SOTA) performance on specific audio understanding tasks but also significantly outperforms existing baselines in prosodic naturalness and emotional alignment, paving the way for the next generation of empathetic AI agents.

Abstract

Recent advancements in Large Audio Language Models (LALMs) have demonstrated exceptional performance in speech recognition and translation. However, existing models often suffer from a disconnect between perception and expression, resulting in a robotic "read-speech" style that lacks the spontaneity and emotional resonance of real human interaction. In this report, we introduce Hello-Chat, an end-to-end audio language model designed for realistic social scenarios. By leveraging a massive dataset of real-life conversations and employing a modality-interleaved training strategy, Hello-Chat achieves a breakthrough in anthropomorphic generation. Experimental results show that our model not only reaches state-of-the-art (SOTA) performance on specific audio understanding tasks but also significantly outperforms existing baselines in prosodic naturalness and emotional alignment, paving the way for the next generation of empathetic AI agents.
Paper Structure (47 sections, 2 figures, 7 tables)

This paper contains 47 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Audio Language Models
  4. Modality Alignment & Instruction Tuning
  5. Text-to-Speech Synthesis
  6. Methodology
  7. Model Architecture
  8. Design of Talker Sequence Modeling
  9. Token-based Acoustic Control
  10. Context-aware Sequence Modeling
  11. Design for Caption Data Construction
  12. Design for Multi-Modal Instruction Following
  13. Design for Modality Alignment Training Strategy
  14. Design for Capability Preservation via Distillation
  15. Training
  16. ...and 32 more sections

Figures (2)

  • Figure 1: Architecture of Hello-Chat.
  • Figure 2: Token organization strategies for the Talker module. We employ distinct token construction patterns—Dialogue Mode, Long-text Mode, and Standard Sentence Mode—to model different prosodic features and context dependencies.