Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores

TL;DR

MSU-Bench addresses the need for holistic musical score understanding by evaluating LLMs and VLMs on complete scores across textual (ABC notation) and visual (PDF) modalities. It introduces 1,800 generative QA pairs spanning four hierarchical levels of musical comprehension, with data drawn from 150 scores and manual answer validation, enabling controlled, multimodal reasoning. Zero-shot results reveal a pronounced textual–visual modality gap and fragility of sustained multi-level reasoning, while LoRA-based fine-tuning substantially boosts performance and preserves general knowledge. Overall, MSU-Bench provides a rigorous, multimodal testbed for advancing AI-driven musicology and symbolic-music understanding beyond fragmentary or multiple-choice tasks.

Abstract

Understanding complete musical scores requires reasoning over symbolic structures such as pitch, rhythm, harmony, and form. Despite the rapid progress of Large Language Models (LLMs) and Vision-Language Models (VLMs) in natural language and multimodal tasks, their ability to comprehend musical notation remains underexplored. We introduce Musical Score Understanding Benchmark (MSU-Bench), the first large-scale, human-curated benchmark for evaluating score-level musical understanding across both textual (ABC notation) and visual (PDF) modalities. MSU-Bench comprises 1,800 generative question-answer (QA) pairs drawn from works spanning Bach, Beethoven, Chopin, Debussy, and others, organised into four progressive levels of comprehension: Onset Information, Notation & Note, Chord & Harmony, and Texture & Form. Through extensive zero-shot and fine-tuned evaluations of over 15+ state-of-the-art (SOTA) models, we reveal sharp modality gaps, fragile level-wise success rates, and the difficulty of sustaining multilevel correctness. Fine-tuning markedly improves performance in both modalities while preserving general knowledge, establishing MSU-Bench as a rigorous foundation for future research at the intersection of Artificial Intelligence (AI), musicological, and multimodal reasoning.

Figures (8)

• Figure 1: (a) Hallucination. When queried about specific score features in bars, VLMs often fabricate responses that are not grounded in the actual score. (b) Ideal scenario. Models should accurately localise and analyse bars, thereby supporting reliable higher-level musicological reasoning.
• Figure 2: Illustration of multi-level score understanding in MSU-Bench using Mussorgsky’s Pictures at an Exhibition. (a) Raw score excerpt with annotated tasks across four levels of difficulty. (b) Metadata encoded in ABC notation. (c) Musical content represented in ABC notation, including voices and chord structures. (d) Sample questions for each level, demonstrating progression from foundational concepts to higher-level musical reasoning.
• Figure 3: Distribution of 4-Level Questions.
• Figure 4: Level-wise Success Rate. We evaluate model performance under textual and visual QA. Numbers below each figure show how many scores remain answerable after each level.
• Figure 5: Frequency distribution of composers represented in MSU-Bench. The histogram illustrates the number of pieces per composer, with Franz Schubert, Liszt, Mendelssohn, and Edvard Grieg appearing most frequently, while representation gradually decreases for others.