MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio

TL;DR

MuseAgent tackles the challenge of grounding multimodal music understanding by integrating a measure-wise optical music recognition front-end and an automatic music transcription pipeline with a retrieval-augmented, memory-enabled LLM controller. The proposed MuseBench provides a holistic, agent-focused benchmark spanning music theory, score interpretation, and performance analysis across text, image, and audio, with piano repertoire as a testbed. Empirical results show that the agent-based MuseAgent significantly outperforms broad MLLMs and monolithic systems on both image and audio tasks, and that structured perceptual grounding (M-OMR and AMT) is crucial for fine-grained symbolic and expressive understanding. The work demonstrates the practical value of combining structured perceptual modules, retrieval, and memory for interactive, cross-modal music reasoning, and offers a extensible framework for future music analysis, education, and archival applications.

Abstract

Despite recent advances in multimodal large language models (MLLMs), their ability to understand and interact with music remains limited. Music understanding requires grounded reasoning over symbolic scores and expressive performance audio, which general-purpose MLLMs often fail to handle due to insufficient perceptual grounding. We introduce MuseAgent, a music-centric multimodal agent that augments language models with structured symbolic representations derived from sheet music images and performance audio. By integrating optical music recognition and automatic music transcription modules, MuseAgent enables multi-step reasoning and interaction over fine-grained musical content. To systematically evaluate music understanding capabilities, we further propose MuseBench, a benchmark covering music theory reasoning, score interpretation, and performance-level analysis across text, image, and audio modalities. Experiments show that existing MLLMs perform poorly on these tasks, while MuseAgent achieves substantial improvements, highlighting the importance of structured multimodal grounding for interactive music understanding.

Figures (8)

• Figure 1: Overview of MuseBench and MuseAgent. MuseBench consists of multimodal music understanding tasks across text, image, and audio modalities, covering music theory, sheet score analysis, performance interpretation. MuseAgent integrates these modalities via sheet symbolic recognition, audio alignment, and retrieval modules, enabling large language models to answer complex music questions.
• Figure 2: The MuseAgent framework integrates M-OMR, AMT, and music retrieval (explicit/implicit) into a unified large language model (LLM)-based system. Each perceptual module converts raw multimodal inputs into structured symbolic representations (e.g., ABC, MusicXML, JSON), which are incorporated into a retrieval-augmented generation (RAG) pipeline. The LLM acts as an agentic controller that dynamically orchestrates module usage depending on user intent, while a memory bank supports multi-turn dialogue and retrieval of prior outputs.
• Figure 3: Comparison between (A) NotaGPT, which performs note-level segmentation with frozen visual and text encoders, and (B) our proposed Measure-wise OMR approach. The flame symbol denotes trainable modules, while the snowflake symbol indicates frozen components.
• Figure 4: Distribution of questions in MuseBench. It consists of 28 task types across three modalities. Tasks are relatively evenly distributed to ensure balanced evaluation.
• Figure 5: Image-to-ABC Conversion Comparison. Results are evaluated on the standardized benchmark introduced in NotaGPT tang2025notamultimodalmusicnotation.