LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection

TL;DR

LadderSym tackles music practice error detection by jointly processing audio references and practice recordings through a two-stream Ladder encoder with inter-stream alignment. It also leverages symbolic scores as prompts to the decoder to reduce score ambiguity. The method achieves state-of-the-art F1 on MAESTRO-E and CocoChorales-E, with large gains in missed and extra note detection, especially in high-concurrency piano pieces. The work provides transferable insights into cross-modal evaluation and alignment, with potential applications in reinforcement learning and human-skill assessment.

Abstract

Music learners can greatly benefit from tools that accurately detect errors in their practice. Existing approaches typically compare audio recordings to music scores using heuristics or learnable models. This paper introduces \textit{LadderSym}, a novel Transformer-based method for music error detection. \textit{LadderSym} is guided by two key observations about the state-of-the-art approaches: (1) late fusion limits inter-stream alignment and cross-modality comparison capability; and (2) reliance on score audio introduces ambiguity in the frequency spectrum, degrading performance in music with concurrent notes. To address these limitations, \textit{LadderSym} introduces (1) a two-stream encoder with inter-stream alignment modules to improve audio comparison capabilities and error detection F1 scores, and (2) a multimodal strategy that leverages both audio and symbolic scores by incorporating symbolic representations as decoder prompts, reducing ambiguity and improving F1 scores. We evaluate our method on the \textit{MAESTRO-E} and \textit{CocoChorales-E} datasets by measuring the F1 score for each note category. Compared to the previous state of the art, \textit{LadderSym} more than doubles F1 for missed notes on \textit{MAESTRO-E} (26.8\% $\rightarrow$ 56.3\%) and improves extra note detection by 14.4 points (72.0\% $\rightarrow$ 86.4\%). Similar gains are observed on \textit{CocoChorales-E}. This work introduces general insights about comparison models that could inform sequence evaluation tasks for reinforcement Learning, human skill assessment, and model evaluation.

Figures (8)

• Figure 1: The error detection task for music practice. The left music score represents the reference, while the right music score is the practice audio transcription. Solutions must detect three types of errors: extra notes, where a note (e.g., "G") is played but not in the reference; missed notes, where a reference note is omitted (e.g., "E" is not played); and wrong notes, where a missed note and an extra note coincide (e.g., playing "B" instead of the expected "C").
• Figure 2: (a) Explicit alignment methods align the score with audio and compare it to the transcribed practice benetos_score-informed_2012wang_identifying_2017. (b) Latent alignment methods synthesize the score to audio and pass it to the encoder (enc) directly, without explicit alignment chou_detecting_2025. (c) Our method, LadderSym, is a latent alignment approach that incorporates symbolic score prompting to address score ambiguity and introduces cross-attention modules to enhance cross-stream information flow. The asymmetric alignment enables each stream to specialize their feature extraction, reducing redundancy and decoupling feature extraction.
• Figure 3: Architecture of LadderSym. We feed both score audio and practice audio into Ladder, our novel encoder with inter-stream alignment modules. Their latent features are concatenated and used as context for the autoregressive decoder. This is done via cross-attention between the encoder outputs and the decoder inputs. To create LadderSym, we prepend a symbolic prompt that is generated from a MIDI version of the score before the start-of-sequence token to provide a different representation of the reference score. The T5 decoder then produces MIDI-like tokens, labeling notes as correct, missed, or extra.
• Figure 4: Similarity between (a) Dynamic Time Warping and (b) Learned alignment patterns in the alignment module.
• Figure 5: Topology of the Encoder Block. Alignment modules alternate between streams, allowing iterative alignment and fusion of information from the score and practice audio. The encoder blocks process the intermediate representations.