Do Foundational Audio Encoders Understand Music Structure?
Keisuke Toyama, Zhi Zhong, Akira Takahashi, Shusuke Takahashi, Yuki Mitsufuji
TL;DR
The paper evaluates 11 foundational audio encoders (FAEs) across learning paradigms to assess their ability to support music structure analysis (MSA). Using a simple linear backend on Harmonix and RWC-pop, it finds that self-supervised masked language modeling (MLM) FAEs trained on music data with longer contexts (e.g., MusicFM, MERT) outperform other FAEs on boundary detection and function prediction. Supervised FAEs lag unless fine-tuned, and tokenization-based SSL methods underperform compared to MLM. The results highlight the importance of long-term context and semantic representations for MSA and suggest MLM FAEs as strong backbones for long-form music tasks and evaluation metrics.
Abstract
In music information retrieval (MIR) research, the use of pretrained foundational audio encoders (FAEs) has recently become a trend. FAEs pretrained on large amounts of music and audio data have been shown to improve performance on MIR tasks such as music tagging and automatic music transcription. However, their use for music structure analysis (MSA) remains underexplored. Although many open-source FAE models are available, only a small subset has been examined for MSA, and the impact of factors such as learning methods, training data, and model context length on MSA performance remains unclear. In this study, we conduct comprehensive experiments on 11 types of FAEs to investigate how these factors affect MSA performance. Our results demonstrate that FAEs using selfsupervised learning with masked language modeling on music data are particularly effective for MSA. These findings pave the way for future research in MSA.