SAR-LM: Symbolic Audio Reasoning with Large Language Models
Termeh Taheri, Yinghao Ma, Emmanouil Benetos
TL;DR
SAR-LM addresses the challenge of audio reasoning with large language models by replacing opaque dense embeddings with symbolic, time-aligned audio features (transcripts, emotions, sound-event tags, notes, and chords). The pipeline builds interpretable prompts, with an optional captioning step, and uses LLMs to perform reasoning, complemented by a GPT-style agent for dynamic feature selection. Across MMAU, MMAR, and OmniBench, SAR-LM achieves competitive accuracy while enabling detailed error analysis that links failures to specific symbolic components, thus improving interpretability without sacrificing performance. This approach offers a practical path toward transparent, debuggable audio reasoning and points to future enhancements via unified feature backends and stronger encoders to further boost performance.
Abstract
Large language models (LLMs) have advanced in text and vision, but their reasoning on audio remains limited. Most existing methods rely on dense audio embeddings, which are difficult to interpret and often fail on structured reasoning tasks. Caption-based approaches, introduced in recent benchmarks such as MMAU, improve performance by translating audio into text, yet still depend on dense embeddings as input, offering little insight when models fail. We present SAR-LM, a symbolic audio reasoning pipeline that builds on this caption-based paradigm by converting audio into structured, human-readable features across speech, sound events, and music. These symbolic inputs support both reasoning and transparent error analysis, enabling us to trace failures to specific features. Across three benchmarks, MMAU, MMAR, and OmniBench, SAR-LM achieves competitive results, while prioritizing interpretability as its primary contribution.