OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia

TL;DR

OSUM introduces an open-source Speech Understanding Language Model that operates under academic resource constraints by fusing a Whisper-based encoder with a LoRA-finetuned Qwen2 LLM and adopting an ASR+X multitask training strategy. The approach expands Whisper with additional multi-task instructions, uses a hybrid adaptor, and tailors data pipelines across eight speech tasks, including STTC and SSR, with meticulous data curation and labeling methods. Experimental results demonstrate OSUM's competitive performance against industry models while using substantially less data and compute, and highlight benefits in cross-task generalization and code-switching scenarios. By emphasizing transparency in data processing and training, OSUM aims to democratize access to advanced speech understanding research and provide a practical blueprint for academia. The work also outlines future directions toward multi-task on a single inference, multilingual capabilities, and full-duplex interactions, reinforcing the value of open science in SULM development.

Abstract

Large Language Models (LLMs) have made significant progress in various downstream tasks, inspiring the development of Speech Understanding Language Models (SULMs) to enable comprehensive speech-based interactions. However, most advanced SULMs are developed by the industry, leveraging large-scale datasets and computational resources that are not readily available to the academic community. Moreover, the lack of transparency in training details creates additional barriers to further innovation. In this study, we present OSUM, an Open Speech Understanding Model designed to explore the potential of training SLUMs under constrained academic resources. The OSUM model combines a Whisper encoder with a Qwen2 LLM and supports a wide range of speech tasks, including speech recognition (ASR), speech recognition with timestamps (SRWT), vocal event detection (VED), speech emotion recognition (SER), speaking style recognition (SSR), speaker gender classification (SGC), speaker age prediction (SAP), and speech-to-text chat (STTC). By employing an ASR+X training strategy, OSUM achieves efficient and stable multi-task training by simultaneously optimizing ASR alongside target tasks. Beyond delivering strong performance, OSUM emphasizes transparency by providing openly available data preparation and training methodologies, offering valuable insights and practical guidance for the academic community. By doing so, we aim to accelerate research and innovation in advanced SULM technologies.

Figures (2)

• Figure 1: Comparison of Qwen2-Audio and our OSUM model. In most tasks, OSUM achieves a better performance than Qwen2-Audio despite using significantly fewer computational resources and training data. The values for each model's task in the radar chart are based on the average results on the public and internal test sets, as shown in Table \ref{['tab:res_asr']} and Table \ref{['tab:res_multi']}.
• Figure 2: The overview of the architecture and tasks of OSUM.