Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio

TL;DR

This work proposes a unified cross-modal translation framework for score images, symbolic music, MIDI, and audio by training a single Transformer on multiple translation tasks with a shared token space. A large-scale YouTube Score Video (YTSV) dataset and a discretized, multimodal tokenization pipeline (RQVAE for images, DAC for audio, Linearized MusicXML, and MIDI-Like tokens) enable end-to-end I2A and A2I translation. The approach yields state-of-the-art results in OMR, enables the first end-to-end score-image–conditioned audio generation, and demonstrates performance gains across AMT and related tasks through multitask learning. These results suggest significant practical impact for scalable, integrated music understanding and generation across diverse representations.

Abstract

Music exists in various modalities, such as score images, symbolic scores, MIDI, and audio. Translations between each modality are established as core tasks of music information retrieval, such as automatic music transcription (audio-to-MIDI) and optical music recognition (score image to symbolic score). However, most past work on multimodal translation trains specialized models on individual translation tasks. In this paper, we propose a unified approach, where we train a general-purpose model on many translation tasks simultaneously. Two key factors make this unified approach viable: a new large-scale dataset and the tokenization of each modality. Firstly, we propose a new dataset that consists of more than 1,300 hours of paired audio-score image data collected from YouTube videos, which is an order of magnitude larger than any existing music modal translation datasets. Secondly, our unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling a single encoder-decoder Transformer to tackle multiple cross-modal translation as one coherent sequence-to-sequence task. Experimental results confirm that our unified multitask model improves upon single-task baselines in several key areas, notably reducing the symbol error rate for optical music recognition from 24.58% to a state-of-the-art 13.67%, while similarly substantial improvements are observed across the other translation tasks. Notably, our approach achieves the first successful score-image-conditioned audio generation, marking a significant breakthrough in cross-modal music generation.

Figures (12)

• Figure 1: Conventional cross-modal conversion tasks in music information retrieval research.
• Figure 2: Overview of our proposed unified multimodal translation framework. We employ a single Transformer encoder-decoder model for each direction— one for Image-to-Audio direction (I2A) tasks and another for Audio-to-Image direction (A2I) tasks. Each model jointly handles multiple translation tasks. All modalities are discretised into token sequences, enabling end-to-end, multitask training entirely at the token level. Note that we train separate models for I2A and A2I directions; the two directions do not share weights.
• Figure 3: The four modalities of music representation used in this paper.
• Figure 4: An example from one of the videos collected for the YouTube Score Video dataset. Slides of sheet music are aligned to the corresponding points in audio.
• Figure 5: Illustration of the music system detection pipeline using the fine-tuned YOLOv8. Music systems detected by fine-tuned YOLOv8 are notated with blue boxes, and detected staff lines are notated with red boxes. Note that the red boxes detect the staff height near clefs, not the clefs themselves.