OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

TL;DR

OpenS2S introduces a fully open-source end-to-end empathetic large speech-language model, addressing opacity and data costs that limit current research. The architecture combines an audio encoder, an instruction-following LLM, a streaming interleaved speech decoder, and a token2wav converter to deliver low-latency, expressive responses. It integrates an automated data-construction pipeline that leverages LLMs and controllable TTS to generate diverse, high-quality empathetic dialogues with minimal human annotation, and provides staged pretraining plus empathetic instruction tuning to avoid overfitting. The authors release all resources, including model weights, training code, and synthetic datasets, to empower the community and advance transparent research in empathetic speech systems.

Abstract

Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details about the architecture, data and development opaque to researchers. Given the critical need for transparent research into the LSLMs and empathetic behavior, we present OpenS2S, a fully open-source, transparent and end-to-end LSLM designed to enable empathetic speech interactions. Based on our empathetic speech-to-text model BLSP-Emo, OpenS2S further employs a streaming interleaved decoding architecture to achieve low-latency speech generation. To facilitate end-to-end training, OpenS2S incorporates an automated data construction pipeline that synthesizes diverse, high-quality empathetic speech dialogues at low cost. By leveraging large language models to generate empathetic content and controllable text-to-speech systems to introduce speaker and emotional variation, we construct a scalable training corpus with rich paralinguistic diversity and minimal human supervision. We release the fully open-source OpenS2S model, including the dataset, model weights, pre-training and fine-tuning codes, to empower the broader research community and accelerate innovation in empathetic speech systems. The project webpage can be accessed at https://casia-lm.github.io/OpenS2S

Figures (4)

• Figure 1: Architecture of our proposed OpenS2S.
• Figure 2: The training process of OpenS2S.
• Figure 3: Automatic workflow for constructing empathetic speech instruction dataset.
• Figure 4: Distributions of emotion, age, and gender in query utterances, and emotion distribution in response utterances.