Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding

Meng Yang, Jon McCormack, Maria Teresa Llano, Wanchao Su, Chao Lei

TL;DR

MIDI-LLaMA addresses the gap in symbolic music understanding by aligning a MIDI encoder (MusicBERT) with the Llama-3-8B language model through a two-stage training pipeline. A scalable annotation pipeline builds a symbolic music–text dataset from GiantMIDI-Piano, enriching it with genre, style, emotion, and expressive intent via GPT-4o and expert validation. Compared to a text-only ABC notation baseline, MIDI-LLaMA achieves superior performance in music captioning and semantic question answering, with human evaluators favoring its music understanding and emotional accuracy. This work demonstrates that incorporating symbolic music encodings into LLMs enhances musical analysis, supports interactive applications, and paves the way for broader symbolic-audio multimodal systems.

Abstract

Recent advances in multimodal large language models (MLLM) for audio music have demonstrated strong capabilities in music understanding, yet symbolic music, a fundamental representation of musical structure, remains unexplored. In this work, we introduce MIDI-LLaMA, the first instruction-following MLLM for symbolic music understanding. Our approach aligns the MIDI encoder MusicBERT and Llama-3-8B via a two-stage pipeline comprising feature alignment and instruction tuning. To support training, we design a scalable annotation pipeline that annotates GiantMIDI-Piano with fine-grained metadata, resulting in a MIDI-text dataset. Compared with the baseline trained on converting MIDI into ABC notation under the same instruction-tuning procedure, MIDI-LLaMA substantially outperforms in captioning and semantic alignment in question answering. Human evaluation further confirms the advantages of MIDI-LLaMA in music understanding, emotion recognition, creativity, and overall preference. These findings demonstrate that incorporating symbolic music into large language models enhances their capacity for musical understanding.

MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding

TL;DR

MIDI-LLaMA addresses the gap in symbolic music understanding by aligning a MIDI encoder (MusicBERT) with the Llama-3-8B language model through a two-stage training pipeline. A scalable annotation pipeline builds a symbolic music–text dataset from GiantMIDI-Piano, enriching it with genre, style, emotion, and expressive intent via GPT-4o and expert validation. Compared to a text-only ABC notation baseline, MIDI-LLaMA achieves superior performance in music captioning and semantic question answering, with human evaluators favoring its music understanding and emotional accuracy. This work demonstrates that incorporating symbolic music encodings into LLMs enhances musical analysis, supports interactive applications, and paves the way for broader symbolic-audio multimodal systems.

Abstract

Recent advances in multimodal large language models (MLLM) for audio music have demonstrated strong capabilities in music understanding, yet symbolic music, a fundamental representation of musical structure, remains unexplored. In this work, we introduce MIDI-LLaMA, the first instruction-following MLLM for symbolic music understanding. Our approach aligns the MIDI encoder MusicBERT and Llama-3-8B via a two-stage pipeline comprising feature alignment and instruction tuning. To support training, we design a scalable annotation pipeline that annotates GiantMIDI-Piano with fine-grained metadata, resulting in a MIDI-text dataset. Compared with the baseline trained on converting MIDI into ABC notation under the same instruction-tuning procedure, MIDI-LLaMA substantially outperforms in captioning and semantic alignment in question answering. Human evaluation further confirms the advantages of MIDI-LLaMA in music understanding, emotion recognition, creativity, and overall preference. These findings demonstrate that incorporating symbolic music into large language models enhances their capacity for musical understanding.
Paper Structure (14 sections, 2 figures, 1 table)

This paper contains 14 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Dataset
  3. Automatic Dataset Annotation Pipeline
  4. Annotation Quality Evaluation
  5. Instruction-Tuning Data Generation
  6. Method
  7. Model Architecture
  8. Training Procedure
  9. Evaluation
  10. Baseline
  11. Task Setting and Evaluation Metrics
  12. Quantitative Results
  13. Human Evaluation on Music Captioning
  14. Conclusion

Figures (2)

  • Figure 1: Overall architecture of MIDI-LLaMA: given a MIDI input, the model can answer music-related questions. The real examples are shown in the graph. Components annotated with the flame symbol are trainable modules.
  • Figure 2: Human comparative evaluation of captions by MIDI-LLaMA and ABC-LLaMA across five dimensions.