Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration

Boyang Wang, Yash Vishe, Xin Xu, Zachary Novack, Julian McAuley, Junda Wu

TL;DR

CSyMR-Bench addresses the gap in symbolic music reasoning benchmarks by providing 126 compositional, multi-step MC questions derived from community discussions and exams, annotated with category and analytical-dimension tags and represented in Humdrum kern notation. It introduces a ReAct-style, music21 tool-augmented agent that splits reasoning into symbolic analyses executed by domain tools and high-level inference by an LLM, achieving consistent accuracy gains over baselines, especially on harmonically complex tasks. The work demonstrates that integrating specialized symbolic analysis tools within an iterative reasoning framework substantially improves reliability over pure prompting, and it highlights both the promise and limitations of current approaches for real-world symbolic-music reasoning. This framework lays groundwork for broader tool-enabled AI in specialized domains, advancing evaluation and development of compositional symbolic reasoning systems in music.

Abstract

Large Language Models (LLMs) are leveraged in symbolic music reasoning, yet existing benchmarks emphasize isolated knowledge or atomic analyses rather than the integrative compositional reasoning needed to connect musical structures. To address this, we present the Compositional Symbolic Music Reasoning Benchmark (CSyMR-Bench), a curated multiple-choice dataset of 126 questions derived from expert forums and professional examinations. Each item involves combining several atomic analyses to arrive at the final answer. Furthermore, we introduce a tool-augmented agent framework that leverages symbolic music analysis tools from the music21 library to address the challenges posed by CSyMR-Bench. Experiments validate that CSyMR-Bench poses a non-trivial challenge across both community-sourced and exam-style questions, while our tool-augmented agent consistently outperforms all baselines, achieving 5-7% absolute accuracy gains.

CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration

TL;DR

CSyMR-Bench addresses the gap in symbolic music reasoning benchmarks by providing 126 compositional, multi-step MC questions derived from community discussions and exams, annotated with category and analytical-dimension tags and represented in Humdrum kern notation. It introduces a ReAct-style, music21 tool-augmented agent that splits reasoning into symbolic analyses executed by domain tools and high-level inference by an LLM, achieving consistent accuracy gains over baselines, especially on harmonically complex tasks. The work demonstrates that integrating specialized symbolic analysis tools within an iterative reasoning framework substantially improves reliability over pure prompting, and it highlights both the promise and limitations of current approaches for real-world symbolic-music reasoning. This framework lays groundwork for broader tool-enabled AI in specialized domains, advancing evaluation and development of compositional symbolic reasoning systems in music.

Abstract

Large Language Models (LLMs) are leveraged in symbolic music reasoning, yet existing benchmarks emphasize isolated knowledge or atomic analyses rather than the integrative compositional reasoning needed to connect musical structures. To address this, we present the Compositional Symbolic Music Reasoning Benchmark (CSyMR-Bench), a curated multiple-choice dataset of 126 questions derived from expert forums and professional examinations. Each item involves combining several atomic analyses to arrive at the final answer. Furthermore, we introduce a tool-augmented agent framework that leverages symbolic music analysis tools from the music21 library to address the challenges posed by CSyMR-Bench. Experiments validate that CSyMR-Bench poses a non-trivial challenge across both community-sourced and exam-style questions, while our tool-augmented agent consistently outperforms all baselines, achieving 5-7% absolute accuracy gains.
Paper Structure (14 sections, 4 figures, 2 tables)

This paper contains 14 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The CSyMR Benchmark
  3. Task Definition
  4. Data Sources and Curation
  5. Categories and Tags
  6. Tool-Augmented Agent
  7. Framework Overview
  8. Implementation Details
  9. Experiments
  10. Setup
  11. Main Results
  12. Case Study
  13. Limitations and Future Work
  14. Conclusion

Figures (4)

  • Figure 1: (a)shows the frequency of each analytical dimension tag. PE, RM, SSS, CH, PI and KS are respectively short for terms Performance & Expression, Rhythm & Meter, Score Structural Statistics, Chord & Harmony, Pitch & Interval and Key & Scale. (b) shows the category distribution of CSyMR-Bench, where CTH Analysis, ER, CST Analysis, CC, EP and GM are short for Complex Tonal-Harmonic Analysis, Editing / Rewriting, Complex Structural-Textural Analysis, Composition & Creative Guidance, Effect / Perceptual Explanation, and Genre / Musician Judgment respectively
  • Figure 2: Representative examples of benchmark questions across different categories. One question is shown for each category; in addition, an excerpt of the corresponding symbolic music for Question 4 is displayed in Humdrum kern format.
  • Figure 3: Overview of the proposed Music21-Tool-Augmented Agent framework. The diagram highlights the Planner, the Thinker (LLM), and the Tooler as the key components of the reasoning loop.
  • Figure 4: Case study of an example question in the Benchmark. The Music21-Augmented Tool Agent gets the correct answer, while CoT fails.