ESPnet-SpeechLM: An Open Speech Language Model Toolkit

TL;DR

This paper presents ESPnet-SpeechLM, an open-source toolkit that democratizes SpeechLM development by reframing speech tasks as universal sequential modeling and providing a modular workflow from preprocessing to evaluation. It introduces task templates, multi-stream tokenization, and flexible backends (ESPnet Transformer or HuggingFace models), along with RLHF options and a HuggingFace-compatible sharing layer. The authors demonstrate scalability with a 1.7B multitask SpeechLM pre-trained on ASR, TTS, TextLM, and AudioLM tasks, using vast datasets (up to 240B text tokens or audio frames) and leveraging a rich evaluation suite (VERSA) across benchmarks, achieving competitive ASR and TTS performance while preserving strong language capabilities. Overall, ESPnet-SpeechLM delivers transparent recipes, reproducible experiments, and practical pathways for building advanced SpeechLMs and voice-driven agents at scale.

Abstract

We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks. The toolkit and its recipes are fully transparent and reproducible at: https://github.com/espnet/espnet/tree/speechlm.

Figures (2)

• Figure 1: The overview of ESPnet-SpeechLM workflow.
• Figure 2: The training sequence $\mathbf{s}^{\text{d}}$ is assembled based on the task template, e.g. single-task ASR as depicted here. The sequence is multi-stream with an extra $n_q$-axis because the codec tokenizes each frame into multiple tokens.