BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning

TL;DR

BASS introduces a comprehensive benchmark to evaluate music understanding and reasoning in audio LMs across four categories—structural segmentation, lyric transcription, musicological analysis, and artist collaboration—through $12$ tasks and 2658 questions drawn from $1993$ songs. The study benchmarks $14$ open-source and frontier models, revealing that while lyric transcription is relatively strong, higher-level musical reasoning, long-range structure localization, and multi-artist analysis remain challenging. A mix of open-ended and multiple-choice formats, along with rigorous evaluation metrics such as $IWER$ and $EMA_N$, enables robust cross-task comparisons. Key findings show that models leverage linguistic priors but struggle with structure, timbre nuances, and temporal localization, highlighting directions for developing audio LMs with deeper, long-form musical understanding and reasoning capabilities. BASS thus serves as a challenging, real-world evaluation framework to guide future research in audio-language modelling for music discovery and analysis.

Abstract

Music understanding is a complex task that often requires reasoning over both structural and semantic elements of audio. We introduce BASS, designed to evaluate music understanding and reasoning in audio language models across four broad categories: structural segmentation, lyric transcription, musicological analysis, and artist collaboration. BASS comprises 2658 questions spanning 12 tasks, 1993 unique songs and covering over 138 hours of music from a wide range of genres and tracks, crafted to assess musicological knowledge and reasoning in real-world scenarios. We evaluate 14 open-source and frontier multimodal LMs, finding that even state-of-the-art models struggle on higher-level reasoning tasks such as structural segmentation and artist collaboration, while performing best on lyric transcription. Our analysis reveals that current models leverage linguistic priors effectively but remain limited in reasoning over musical structure, vocal, and musicological attributes. BASS provides an evaluation framework with widespread applications in music recommendation and search and has the potential to guide the development of audio LMs.

Figures (16)

• Figure 1: Average performance of audio LMs across all task categories in BASS ( the upper bound is 100% in each case). Top-performing models achieve their highest scores on lyric transcription and their lowest on artist collaboration tasks.
• Figure 2: Overview of the tasks included in BASS Benchmark, designed to evaluate audio LMs on musical understanding requiring long-term structure or hierarchical reasoning. It contains twelve tasks across four categories: structural segmentation, structural lyrics transcription, musicological analysis, and artist collaboration.
• Figure 3: Section-wise performance for Structural Segmentation and Lyric Transcription (higher is better). Segmentation performance declines over time, while transcription performance (IWER) is lowest for the Intro compared to Verse and Chorus sections.
• Figure 4: Model accuracy for singing versus rapping vocal styles. Most models identify rapping more accurately than singing.
• Figure 5: Model performance across musicological attributes and genres in the musicological analysis task. (Left) Average performance across specific attributes; while models excel in vocals and lyrics, attributes like rhythm, instrumentation, and sonority remain challenging even for frontier models. (Right) Performance breakdown by genre, showing that models generally perform best on Country and Rock while experiencing a noticeable dip on Hip Hop and Pop.