Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Factual and Musical Evaluation Metrics for Music Language Models

Daniel Chenyu Lin, Michael Freeman, John Thickstun

TL;DR

The paper addresses the misalignment between common NLP metrics and factual understanding in Music LMs. It introduces CLAPText, a musically aware similarity metric based on CLAP embeddings, and a modality-agnostic Factual QA framework that converts open-ended outputs into structured labels for interpretable evaluation using Precision, Recall, and F1. Through Free-Form QA experiments on MusicQA-derived data and Factual QA on FMA and MusicNet, the authors show that traditional metrics often fail to reflect factual audio understanding, while CLAPText and the proposed factual framework provide more reliable assessments and reveal the influence of prompting. The findings suggest that evaluating Music LMs requires metrics that reward factual correctness over linguistic fluency, with broader implications for reliable multimodal language models across domains.

Abstract

Music language models (Music LMs), like vision language models, leverage multimodal representations to answer natural language queries about musical audio recordings. Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct. Specifically, for all Music LMs that we examine, widely-used evaluation metrics such as BLEU, METEOR, and BERTScore fail to measure anything beyond linguistic fluency of the model's responses. To measure the true performance of Music LMs, we propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses. Our framework is agnostic to the modality of the question-answering model and could be generalized to quantify performance in other open-ended question-answering domains. We use open datasets in our experiments and will release all code on publication.

Factual and Musical Evaluation Metrics for Music Language Models

TL;DR

The paper addresses the misalignment between common NLP metrics and factual understanding in Music LMs. It introduces CLAPText, a musically aware similarity metric based on CLAP embeddings, and a modality-agnostic Factual QA framework that converts open-ended outputs into structured labels for interpretable evaluation using Precision, Recall, and F1. Through Free-Form QA experiments on MusicQA-derived data and Factual QA on FMA and MusicNet, the authors show that traditional metrics often fail to reflect factual audio understanding, while CLAPText and the proposed factual framework provide more reliable assessments and reveal the influence of prompting. The findings suggest that evaluating Music LMs requires metrics that reward factual correctness over linguistic fluency, with broader implications for reliable multimodal language models across domains.

Abstract

Music language models (Music LMs), like vision language models, leverage multimodal representations to answer natural language queries about musical audio recordings. Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct. Specifically, for all Music LMs that we examine, widely-used evaluation metrics such as BLEU, METEOR, and BERTScore fail to measure anything beyond linguistic fluency of the model's responses. To measure the true performance of Music LMs, we propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses. Our framework is agnostic to the modality of the question-answering model and could be generalized to quantify performance in other open-ended question-answering domains. We use open datasets in our experiments and will release all code on publication.

Paper Structure

This paper contains 22 sections, 1 equation, 1 figure, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Free-Form Question Answering
  4. Baseline
  5. Skyline (Paraphrase)
  6. Adversarial
  7. CLAPText
  8. Results
  9. Factual Question Answering
  10. Keyword Extraction
  11. Evaluation Metrics for Keyword Labels
  12. Results
  13. Conclusion
  14. More Details on Music LMs
  15. Full Factuality Experiment
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Overview of our evaluation methodology. Top: Given (question, audio) pairs, we study the behavior of open-ended text metrics by comparing reference answers to outputs of (1. Correct; blue) the Music LM, provided with the intended audio for the corresponding question; (2. Baseline; orange) the Music LM, provided an audio input chosen at random from the dataset; (3. Skyline; yellow) an LLM, asked to paraphrase the reference text; no Music LM should be able to outperform this skyline result (4. Adversarial; green) an LLM, asked to make subtle edits to the reference text that completely change its meaning. Bottom: Our factuality framework for converting a labeled dataset into a benchmark for Music LMs. A Music LM first predicts open-ended text in response to a prompt for factual information. A large language model then performs keyword extraction under strict rules to canonicalize this free-form response into structured labels. These extracted labels are compared to ground-truth labels to compute factuality metrics such as precision, recall, and F1-score, enabling direct, interpretable evaluation of factual correctness.