Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Musically Informed Evaluation of Piano Transcription Models

Patricia Hu, Lukáš Samuel Marták, Carlos Cancino-Chacón, Gerhard Widmer

TL;DR

The paper addresses the mismatch between standard IR metrics and musically meaningful transcription quality in solo piano AMT, and the poor generalization of models to real-world data. It introduces four musically informed metrics—Timing, Articulation, Harmony, and Dynamics—computed from ground-truth MIDI and model outputs using skyline-based voice separation and spiral-array harmony measures. The study evaluates three contemporary piano transcription models (OaF, T5, Kong) on a MAESTRO subset, demonstrating that the new metrics reveal expressive-structure errors and generalization weaknesses not captured by traditional IR metrics. The work provides data and code to promote interpretable evaluation for expressive piano transcription and guides future extensions toward score alignment and perceptual validation.

Abstract

Automatic piano transcription models are typically evaluated using simple frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics do not provide insights into the transcription quality of specific musical aspects such as articulation, dynamics, or rhythmic precision of the output, which are essential in the context of expressive performance analysis. Furthermore, in recent years, MAESTRO has become the de-facto training and evaluation dataset for such models. However, inference performance has been observed to deteriorate substantially when applied on out-of-distribution data, thereby questioning the suitability and reliability of transcribed outputs from such models for specific MIR tasks. In this work, we investigate the performance of three state-of-the-art piano transcription models in two experiments. In the first one, we propose a variety of musically informed evaluation metrics which, in contrast to the IR metrics, offer more detailed insight into the musical quality of the transcriptions. In the second experiment, we compare inference performance on real-world and perturbed audio recordings, and highlight musical dimensions which our metrics can help explain. Our experimental results highlight the weaknesses of existing piano transcription metrics and contribute to a more musically sound error analysis of transcription outputs.

Towards Musically Informed Evaluation of Piano Transcription Models

TL;DR

The paper addresses the mismatch between standard IR metrics and musically meaningful transcription quality in solo piano AMT, and the poor generalization of models to real-world data. It introduces four musically informed metrics—Timing, Articulation, Harmony, and Dynamics—computed from ground-truth MIDI and model outputs using skyline-based voice separation and spiral-array harmony measures. The study evaluates three contemporary piano transcription models (OaF, T5, Kong) on a MAESTRO subset, demonstrating that the new metrics reveal expressive-structure errors and generalization weaknesses not captured by traditional IR metrics. The work provides data and code to promote interpretable evaluation for expressive piano transcription and guides future extensions toward score alignment and perceptual validation.

Abstract

Automatic piano transcription models are typically evaluated using simple frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics do not provide insights into the transcription quality of specific musical aspects such as articulation, dynamics, or rhythmic precision of the output, which are essential in the context of expressive performance analysis. Furthermore, in recent years, MAESTRO has become the de-facto training and evaluation dataset for such models. However, inference performance has been observed to deteriorate substantially when applied on out-of-distribution data, thereby questioning the suitability and reliability of transcribed outputs from such models for specific MIR tasks. In this work, we investigate the performance of three state-of-the-art piano transcription models in two experiments. In the first one, we propose a variety of musically informed evaluation metrics which, in contrast to the IR metrics, offer more detailed insight into the musical quality of the transcriptions. In the second experiment, we compare inference performance on real-world and perturbed audio recordings, and highlight musical dimensions which our metrics can help explain. Our experimental results highlight the weaknesses of existing piano transcription metrics and contribute to a more musically sound error analysis of transcription outputs.
Paper Structure (14 sections, 1 equation, 4 figures, 3 tables)

This paper contains 14 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Musically informed metrics
  4. Timing
  5. Articulation
  6. Harmony
  7. Dynamics
  8. Data
  9. Demonstration of musically informed metrics
  10. Out-of-distribution inference
  11. Generalization on real-world recordings
  12. Evaluation on perturbed audio recordings
  13. Conclusion
  14. Acknowledgments

Figures (4)

  • Figure 1: Model performance comparison as evaluated on note-offset F1 score and our proposed musical metrics, by composer.
  • Figure 2: Relationship between model performance evaluated on note-offset-velocity F1 score and our proposed dynamics measure.
  • Figure 3: Performance degradation measured by note-offset F1, Melody IOI and Cloud Momentum metrics.
  • Figure :