Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network

Zhanhong He, Hanyu Meng, David Huang, Roberto Togneri

TL;DR

This work tackles piano dynamics estimation from audio by introducing a compact, end-to-end multi-task network that jointly predicts dynamics, change points, beats, and downbeats. The model uses Bark-scale loudness (BSSL) as input and a three-branch multi-scale encoder with an 8-expert Mixture-of-Experts to share a latent representation across tasks, achieving state-of-the-art results on dynamics and change-point detection on MazurkaBL. It demonstrates substantial parameter efficiency and long-context processing benefits, enabling scalable analysis of musical expression. The approach paves the way for end-to-end pipelines that can generate scores with dynamic markings directly from performance audio.

Abstract

Estimating piano dynamic from audio recordings is a fundamental challenge in computational music analysis. In this paper, we propose an efficient multi-task network that jointly predicts dynamic levels, change points, beats, and downbeats from a shared latent representation. These four targets form the metrical structure of dynamics in the music score. Inspired by recent vocal dynamic research, we use a multi-scale network as the backbone, which takes Bark-scale specific loudness as the input feature. Compared to log-Mel as input, this reduces model size from 14.7 M to 0.5 M, enabling long sequential input. We use a 60-second audio length in audio segmentation, which doubled the length of beat tracking commonly used. Evaluated on the public MazurkaBL dataset, our model achieves state-of-the-art results across all tasks. This work sets a new benchmark for piano dynamic estimation and delivers a powerful and compact tool, paving the way for large-scale, resource-efficient analysis of musical expression.

Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network

TL;DR

This work tackles piano dynamics estimation from audio by introducing a compact, end-to-end multi-task network that jointly predicts dynamics, change points, beats, and downbeats. The model uses Bark-scale loudness (BSSL) as input and a three-branch multi-scale encoder with an 8-expert Mixture-of-Experts to share a latent representation across tasks, achieving state-of-the-art results on dynamics and change-point detection on MazurkaBL. It demonstrates substantial parameter efficiency and long-context processing benefits, enabling scalable analysis of musical expression. The approach paves the way for end-to-end pipelines that can generate scores with dynamic markings directly from performance audio.

Abstract

Estimating piano dynamic from audio recordings is a fundamental challenge in computational music analysis. In this paper, we propose an efficient multi-task network that jointly predicts dynamic levels, change points, beats, and downbeats from a shared latent representation. These four targets form the metrical structure of dynamics in the music score. Inspired by recent vocal dynamic research, we use a multi-scale network as the backbone, which takes Bark-scale specific loudness as the input feature. Compared to log-Mel as input, this reduces model size from 14.7 M to 0.5 M, enabling long sequential input. We use a 60-second audio length in audio segmentation, which doubled the length of beat tracking commonly used. Evaluated on the public MazurkaBL dataset, our model achieves state-of-the-art results across all tasks. This work sets a new benchmark for piano dynamic estimation and delivers a powerful and compact tool, paving the way for large-scale, resource-efficient analysis of musical expression.
Paper Structure (15 sections, 2 equations, 2 figures, 2 tables)

This paper contains 15 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Bark-Scale Specific Loudness
  4. Model Architecture
  5. Task-Specific Post-Processing
  6. Loss Function
  7. Experiments
  8. Dataset
  9. Implementation Details
  10. Baselines
  11. Evaluation Metrics
  12. Results
  13. Main Result
  14. Ablation Study
  15. Conclusion

Figures (2)

  • Figure 1: Workflow for adding dynamic markings to an AMT-transcribed score from audio. The highlighted region (90–105 s of Chopin Mazurka Op. 6 No. 1) shows a case study. Dynamic markings are aligned with the musical beat grid, and a change point indicates a contextual shift in the dynamic levels.
  • Figure 2: Architecture of the proposed multi-task framework. A three-branch multi-scale network encodes either log-Mel or BSSL input. Branches operate at lengths $T$, $T/s$, and $T/s^{2}$, with outputs a latent sequence of shape $T\times8$. An 8-expert MMoE with four gates forms task-specific features, followed by linear heads to output four distinct targets. The scaling factor $s$ is a tunable hyperparameter.