Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context

TL;DR

Solla introduces a speech-oriented LLM that hears acoustic context by fusing a Whisper-based audio encoder, an adaptor, an Audio Tagging module, and InternLM2-chat-7b, augmented with ASR-assisted prediction to capture spoken content. The SA-Eval benchmark assesses audio classification, captioning, and QA under easy and hard conditions to reflect real-world acoustic mixtures. Empirical results show Solla matches or surpasses baselines, especially in hard scenarios, and ablation analyses confirm the AT module and ASR-assisted prediction contribute meaningfully to performance. The work advances natural, hands-free interactions with audio-rich environments and provides a dedicated benchmark for evaluating speech-instruction and acoustic-context understanding in LLMs.

Abstract

Large Language Models (LLMs) have recently shown remarkable ability to process not only text but also multimodal inputs such as speech and audio. However, most existing models primarily focus on analyzing input signals using text instructions, overlooking scenarios in which speech instructions and audio are mixed and serve as inputs to the model. To address these challenges, we introduce Solla, a novel framework designed to understand speech-based questions and hear the acoustic context concurrently. Solla incorporates an audio tagging module to effectively identify and represent audio events, as well as an ASR-assisted prediction method to improve comprehension of spoken content. To rigorously evaluate Solla and other publicly available models, we propose a new benchmark dataset called SA-Eval, which includes three tasks: audio event classification, audio captioning, and audio question answering. SA-Eval has diverse speech instruction with various speaking styles, encompassing two difficulty levels, easy and hard, to capture the range of real-world acoustic conditions. Experimental results show that Solla performs on par with or outperforms baseline models on both the easy and hard test sets, underscoring its effectiveness in jointly understanding speech and audio.

Figures (4)

• Figure 1: An illustration of Solla that a speech-oriented LLM can understand the acoustic context and generate the answer.
• Figure 2: The overview framework of Solla. AT module denotes the audio tagging module.
• Figure 3: Evaluation judgment prompt of VGGSound.
• Figure 4: Evaluation judgment prompt of Clotho-AQA.