Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring GPT's Ability as a Judge in Music Understanding

Kun Fang, Ziyu Wang, Gus Xia, Ichiro Fujinaga

TL;DR

This paper investigates the feasibility of using a text-based LLM to judge MIR outputs by converting music to symbolic representations and applying training-free prompt engineering to detect errors in beat tracking, chord extraction, and key estimation. It introduces a concept augmentation framework to test how additional musical knowledge in prompts affects reasoning, and evaluates GPT-3.5 across three MIR tasks. Results indicate the LLM outperforms random baselines and benefits from richer musical concepts, establishing a baseline for future LLM-based MIR research. The approach is training-free and scalable, though it encounters limitations such as output randomness and hallucination, guiding future work toward real MIR errors and potential fine-tuning.

Abstract

Recent progress in text-based Large Language Models (LLMs) and their extended ability to process multi-modal sensory data have led us to explore their applicability in addressing music information retrieval (MIR) challenges. In this paper, we use a systematic prompt engineering approach for LLMs to solve MIR problems. We convert the music data to symbolic inputs and evaluate LLMs' ability in detecting annotation errors in three key MIR tasks: beat tracking, chord extraction, and key estimation. A concept augmentation method is proposed to evaluate LLMs' music reasoning consistency with the provided music concepts in the prompts. Our experiments tested the MIR capabilities of Generative Pre-trained Transformers (GPT). Results show that GPT has an error detection accuracy of 65.20%, 64.80%, and 59.72% in beat tracking, chord extraction, and key estimation tasks, respectively, all exceeding the random baseline. Moreover, we observe a positive correlation between GPT's error finding accuracy and the amount of concept information provided. The current findings based on symbolic music input provide a solid ground for future LLM-based MIR research.

Exploring GPT's Ability as a Judge in Music Understanding

TL;DR

This paper investigates the feasibility of using a text-based LLM to judge MIR outputs by converting music to symbolic representations and applying training-free prompt engineering to detect errors in beat tracking, chord extraction, and key estimation. It introduces a concept augmentation framework to test how additional musical knowledge in prompts affects reasoning, and evaluates GPT-3.5 across three MIR tasks. Results indicate the LLM outperforms random baselines and benefits from richer musical concepts, establishing a baseline for future LLM-based MIR research. The approach is training-free and scalable, though it encounters limitations such as output randomness and hallucination, guiding future work toward real MIR errors and potential fine-tuning.

Abstract

Recent progress in text-based Large Language Models (LLMs) and their extended ability to process multi-modal sensory data have led us to explore their applicability in addressing music information retrieval (MIR) challenges. In this paper, we use a systematic prompt engineering approach for LLMs to solve MIR problems. We convert the music data to symbolic inputs and evaluate LLMs' ability in detecting annotation errors in three key MIR tasks: beat tracking, chord extraction, and key estimation. A concept augmentation method is proposed to evaluate LLMs' music reasoning consistency with the provided music concepts in the prompts. Our experiments tested the MIR capabilities of Generative Pre-trained Transformers (GPT). Results show that GPT has an error detection accuracy of 65.20%, 64.80%, and 59.72% in beat tracking, chord extraction, and key estimation tasks, respectively, all exceeding the random baseline. Moreover, we observe a positive correlation between GPT's error finding accuracy and the amount of concept information provided. The current findings based on symbolic music input provide a solid ground for future LLM-based MIR research.
Paper Structure (15 sections, 1 equation, 3 figures, 2 tables)

This paper contains 15 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Definition and Data Representation
  5. Beat Tracking Error Detection
  6. Chord Extraction Error Detection
  7. Key Estimation Error Detection
  8. Prompt Structure
  9. Concept Augmentation
  10. Experiments
  11. Datasets
  12. Evaluation Metrics
  13. Evaluation Results
  14. Conclusion and Future Work
  15. Acknowledgments

Figures (3)

  • Figure 1: The example prompts and model outputs for the three error d etection MIR tasks: beat tracking, chord extraction, and key estimation. Some keywords are highlighted in red in this figure for better readability. Orange texts indicate omitted content. The prompt structure is shown on the left.
  • Figure 2: The impact of concept augmentation on GPT's behavior in three MIR error detection tasks: 1) Basic Concepts (left), 2) Concept Introduction (middle), and 3) Concept Masking: all music domain concepts removed (right). Red color indicates the basic concepts. Pink color indicates the introduced concepts. Purple color represents the expression after masking all music-related concepts. Underlines denote reasoning process. The checkmark indicates a correct judgment made by GPT, while the cross indicates an incorrect judgment by GPT.
  • Figure :