Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Multimodal Large Language Models on Core Music Perception Tasks

Brandon James Carone, Iran R. Roman, Pablo Ripollés

TL;DR

This work interrogates whether multimodal large language models genuinely understand music or rely on surface cues by evaluating three core music perception tasks (Syncopation Scoring, Transposition Detection, Chord Quality Identification) with three SOTA models. It introduces a neuro-symbolic adaptation of LogicLM to separate perceptual input from symbolic reasoning, and employs a factorial design across modality (audio vs MIDI), reasoning strategy (Standalone, CoT, LogicLM), and shot setting (zero- vs few-shot). Results reveal a clear perceptual gap: MIDI data yields near-ceiling accuracy while audio input produces substantial drops, indicating a bottleneck in audio perception; reasoning strategies provide limited gains, and LogicLM remains brittle on audio despite strong MIDI performance. The findings underscore the need for robust audio-first front-ends and principled handling of uncertainty in downstream symbolic solvers, offering actionable guidance for building audio-native music analysis systems.

Abstract

Multimodal Large Language Models (LLMs) claim "musical understanding" via evaluations that conflate listening with score reading. We benchmark three SOTA LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni) across three core music skills: Syncopation Scoring, Transposition Detection, and Chord Quality Identification. Moreover, we separate three sources of variability: (i) perceptual limitations (audio vs. MIDI inputs), (ii) exposure to examples (zero- vs. few-shot manipulations), and (iii) reasoning strategies (Standalone, CoT, LogicLM). For the latter we adapt LogicLM, a framework combining LLMs with symbolic solvers to perform structured reasoning, to music. Results reveal a clear perceptual gap: models perform near ceiling on MIDI but show accuracy drops on audio. Reasoning and few-shot prompting offer minimal gains. This is expected for MIDI, where performance reaches saturation, but more surprising for audio, where LogicLM, despite near-perfect MIDI accuracy, remains notably brittle. Among models, Gemini Pro achieves the highest performance across most conditions. Overall, current systems reason well over symbols (MIDI) but do not yet "listen" reliably from audio. Our method and dataset make the perception-reasoning boundary explicit and offer actionable guidance for building robust, audio-first music systems.

Evaluating Multimodal Large Language Models on Core Music Perception Tasks

TL;DR

This work interrogates whether multimodal large language models genuinely understand music or rely on surface cues by evaluating three core music perception tasks (Syncopation Scoring, Transposition Detection, Chord Quality Identification) with three SOTA models. It introduces a neuro-symbolic adaptation of LogicLM to separate perceptual input from symbolic reasoning, and employs a factorial design across modality (audio vs MIDI), reasoning strategy (Standalone, CoT, LogicLM), and shot setting (zero- vs few-shot). Results reveal a clear perceptual gap: MIDI data yields near-ceiling accuracy while audio input produces substantial drops, indicating a bottleneck in audio perception; reasoning strategies provide limited gains, and LogicLM remains brittle on audio despite strong MIDI performance. The findings underscore the need for robust audio-first front-ends and principled handling of uncertainty in downstream symbolic solvers, offering actionable guidance for building audio-native music analysis systems.

Abstract

Multimodal Large Language Models (LLMs) claim "musical understanding" via evaluations that conflate listening with score reading. We benchmark three SOTA LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni) across three core music skills: Syncopation Scoring, Transposition Detection, and Chord Quality Identification. Moreover, we separate three sources of variability: (i) perceptual limitations (audio vs. MIDI inputs), (ii) exposure to examples (zero- vs. few-shot manipulations), and (iii) reasoning strategies (Standalone, CoT, LogicLM). For the latter we adapt LogicLM, a framework combining LLMs with symbolic solvers to perform structured reasoning, to music. Results reveal a clear perceptual gap: models perform near ceiling on MIDI but show accuracy drops on audio. Reasoning and few-shot prompting offer minimal gains. This is expected for MIDI, where performance reaches saturation, but more surprising for audio, where LogicLM, despite near-perfect MIDI accuracy, remains notably brittle. Among models, Gemini Pro achieves the highest performance across most conditions. Overall, current systems reason well over symbols (MIDI) but do not yet "listen" reliably from audio. Our method and dataset make the perception-reasoning boundary explicit and offer actionable guidance for building robust, audio-first music systems.
Paper Structure (23 sections, 1 figure, 3 tables)

This paper contains 23 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Tasks
  4. Syncopation Scoring
  5. Transposition Detection
  6. Chord Quality Identification
  7. Stimuli
  8. Implementation
  9. Standalone, CoT, LogicLM.
  10. Zero-shot vs. few-shot.
  11. Per-task modularity.
  12. Audio vs. MIDI modality.
  13. Models and inference environment
  14. Results
  15. Overall performance across modalities.
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Model accuracy across modalities and prompting strategies. Error bars indicate standard deviation and significance is based on Welch's t-tests carried out with SciPy. Figure 1A shows the collapsed (Shot and Prompting Method) mean accuracy (%) for Gemini 2.5 Flash, Gemini 2.5 Pro, and Qwen 2.5-Omni across Audio (dark grey) and Symbolic (MIDI) (light grey) inputs. Both Gemini models performed significantly better on MIDI, and Qwen showed a similar trend. Figure 1B shows the collapsed (Model and Prompting Method) overall model accuracy by task (Syncopation, Transposition, Chord Quality) under Zeroshot (dark grey) and Fewshot (light grey) prompting. No significant effects of Shot were exhibited.