Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenMU: Your Swiss Army Knife for Music Understanding

Mengjie Zhao, Zhi Zhong, Zhuoyuan Mao, Shiqi Yang, Wei-Hsiang Liao, Shusuke Takahashi, Hiromi Wakaki, Yuki Mitsufuji

TL;DR

Using OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music, the OpenMU model, outperforms baseline models such as MU-Llama.

Abstract

We present OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music. To construct OpenMU-Bench, we leveraged existing datasets and bootstrapped new annotations. OpenMU-Bench also broadens the scope of music understanding by including lyrics understanding and music tool usage. Using OpenMU-Bench, we trained our music understanding model, OpenMU, with extensive ablations, demonstrating that OpenMU outperforms baseline models such as MU-Llama. Both OpenMU and OpenMU-Bench are open-sourced to facilitate future research in music understanding and to enhance creative music production efficiency.

OpenMU: Your Swiss Army Knife for Music Understanding

TL;DR

Using OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music, the OpenMU model, outperforms baseline models such as MU-Llama.

Abstract

We present OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music. To construct OpenMU-Bench, we leveraged existing datasets and bootstrapped new annotations. OpenMU-Bench also broadens the scope of music understanding by including lyrics understanding and music tool usage. Using OpenMU-Bench, we trained our music understanding model, OpenMU, with extensive ablations, demonstrating that OpenMU outperforms baseline models such as MU-Llama. Both OpenMU and OpenMU-Bench are open-sourced to facilitate future research in music understanding and to enhance creative music production efficiency.

Paper Structure

This paper contains 20 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Constructing OpenMU-Bench
  4. OpenMU-Bench task types
  5. Individual datasets
  6. Evaluation metrics
  7. Model architecture and training details
  8. Model architecture
  9. Training details
  10. Designing OpenMU and discussions
  11. Number of music tokens
  12. LoRA, task performance, and music information utility
  13. Overall results
  14. Conclusion
  15. Appendix
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Model architecture of OpenMU. In Stage (1), we only tune the music-language projector. In Stage (2), LoRA adapters are added to the LLM and are tuned together with the projector.
  • Figure 2: Training trajectories of Stage (1) (top) and Stage (2) (bottom). The x-axis represents the number of hours elapsed, and the y-axis shows the training loss on a log scale. We vary the number of mean-pooling music tokens from 2 to 128 and experiment with different LoRA parameter combinations, $\alpha/r$. "MovingAvg" represents the moving average.
  • Figure 3: Performance of OpenMU variants on the captioning and reasoning tasks of OpenMU-Bench. For each evaluation metric, such as BLEU, we report the macro average of the model's performance across all OpenMU-Bench subtasks.
  • Figure 4: Left: Performance of OpenMU variants on the captioning and reasoning tasks of OpenMU-Bench using BertScore as the metric. Right: OpenMU performance on MuChoMusic.
  • Figure 5: Converting numerical values (in beats per minute; BPM) of music tempo to natural language descriptions. The conversion is done based on the Italian musical terms.