Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models

Xiaosha Li, Chun Liu, Ziyu Wang

TL;DR

The paper addresses the unreliability of likelihood-based evaluation for music LLMs, showing that absolute loss can decrease when input is perturbed, due to a Context Amnesia Effect. It introduces a noise-injection paradigm and analyzes token-wise loss changes, revealing a characteristic three-stage loss dynamics: a Peak at perturbation onset, an Assimilation plateau during perturbation, and a Recovery phase after perturbation. The findings demonstrate that absolute loss values poorly reflect long-range musical integrity, while the shape and local dynamics of the loss curve encode meaningful information about generated quality. This motivates a profile-based, label-free evaluation framework that leverages loss-curves to assess musical quality and could guide more principled training objectives and sharper benchmarks for music LLMs.

Abstract

The rise of music large language models (LLMs) demands robust methods of evaluating output quality, especially in distinguishing high-quality compositions from "garbage music". Curiously, we observe that the standard cross-entropy loss -- a core training metric -- often decrease when models encounter systematically corrupted music, undermining its validity as a standalone quality indicator. To investigate this paradox, we introduce noise injection experiment, where controlled noise signal of varying lengths are injected into musical contexts. We hypothesize that a model's loss reacting positively to these perturbations, specifically a sharp increase ("Peak" area) for short injection, can serve as a proxy for its ability to discern musical integrity. Experiments with MusicGen models in the audio waveform domain confirm that Music LLMs respond more strongly to local, texture-level disruptions than to global semantic corruption. Beyond exposing this bias, our results highlight a new principle: the shape of the loss curve -- rather than its absolute value -- encodes critical information about the quality of the generated content (i.e., model behavior). We envision this profile-based evaluation as a label-free, model-intrinsic framework for assessing musical quality -- opening the door to more principled training objectives and sharper benchmarks.

When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models

TL;DR

The paper addresses the unreliability of likelihood-based evaluation for music LLMs, showing that absolute loss can decrease when input is perturbed, due to a Context Amnesia Effect. It introduces a noise-injection paradigm and analyzes token-wise loss changes, revealing a characteristic three-stage loss dynamics: a Peak at perturbation onset, an Assimilation plateau during perturbation, and a Recovery phase after perturbation. The findings demonstrate that absolute loss values poorly reflect long-range musical integrity, while the shape and local dynamics of the loss curve encode meaningful information about generated quality. This motivates a profile-based, label-free evaluation framework that leverages loss-curves to assess musical quality and could guide more principled training objectives and sharper benchmarks for music LLMs.

Abstract

The rise of music large language models (LLMs) demands robust methods of evaluating output quality, especially in distinguishing high-quality compositions from "garbage music". Curiously, we observe that the standard cross-entropy loss -- a core training metric -- often decrease when models encounter systematically corrupted music, undermining its validity as a standalone quality indicator. To investigate this paradox, we introduce noise injection experiment, where controlled noise signal of varying lengths are injected into musical contexts. We hypothesize that a model's loss reacting positively to these perturbations, specifically a sharp increase ("Peak" area) for short injection, can serve as a proxy for its ability to discern musical integrity. Experiments with MusicGen models in the audio waveform domain confirm that Music LLMs respond more strongly to local, texture-level disruptions than to global semantic corruption. Beyond exposing this bias, our results highlight a new principle: the shape of the loss curve -- rather than its absolute value -- encodes critical information about the quality of the generated content (i.e., model behavior). We envision this profile-based evaluation as a label-free, model-intrinsic framework for assessing musical quality -- opening the door to more principled training objectives and sharper benchmarks.
Paper Structure (13 sections, 4 equations, 5 figures)

This paper contains 13 sections, 4 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Noise Injection Experiment
  4. Experiment Setting
  5. Experiment Results
  6. Analysis of Loss Dynamics
  7. Token-Wise Loss Dynamics
  8. Validation Through Automated Region Detection
  9. Discussion
  10. Alternative Perturbation: Order Shuffling
  11. Relation to Exposure Bias
  12. Conclusion
  13. Acknowledgment

Figures (5)

  • Figure 1: The figure shows that injecting noise into the input audio can unexpectedly reduce the loss; we refer to this phenomenon as the Context Amnesia Effect.
  • Figure 2: Comparison of model performance under white-noise injection on the dataset, with standard deviation measured across songs.
  • Figure 3: Loss curve of noise injection experiment.
  • Figure 4: Three-stage behavior after noise injection.
  • Figure 5: Example of shuffle order perturbation with experimental results.