Roadmap towards Superhuman Speech Understanding using Large Language Models

TL;DR

A roadmap for advancing speech LLMs is outlined, a benchmark for evaluation is introduced, and key insights into their current limitations and potential are provided, uncovering challenges in using abstract acoustic knowledge and completeness of capability.

Abstract

The success of large language models (LLMs) has prompted efforts to integrate speech and audio data, aiming to create general foundation models capable of processing both textual and non-textual inputs. Recent advances, such as GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves non-semantic information and world knowledge for deeper speech understanding. To guide the development of speech LLMs, we propose a five-level roadmap, ranging from basic automatic speech recognition (ASR) to advanced superhuman models capable of integrating non-semantic information with abstract acoustic knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark, that standardizes critical aspects across various tasks in these five levels, uncovering challenges in using abstract acoustic knowledge and completeness of capability. Our findings reveal gaps in handling paralinguistic cues and abstract acoustic knowledge, and we offer future directions. This paper outlines a roadmap for advancing speech LLMs, introduces a benchmark for evaluation, and provides key insights into their current limitations and potential.

Figures (6)

• Figure 1: Levels of speech understanding using LLMs.
• Figure 2: Cascade and End-to-end paradigms.
• Figure 3: Distribution of three types of training data used by various models
• Figure 4: Representation similarity of different speeches. Each speech pair has the same content but is spoken in a different style. The representation is generated by the Whisper encoder.
• Figure 5: Performance of speech LLMs with different instructions on speaker age task (left) and scene classification task (right). Gray line shows random selection accuracy. Details about the instructions and results are shown in App. \ref{['appendix:prompt']}.