CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

TL;DR

CLaMP 3 presents a universal MIR framework that unifies sheet music, performance signals, audio, and multilingual text in a shared embedding space via contrastive learning and multi-stage alignment. It introduces M4-RAG to generate 2.31 million multilingual music-text examples and WikiMT-X as a holistic 1,000-triplet benchmark, both publicly released to foster research. Empirical results show state-of-the-art performance across symbolic and audio retrieval, with strong cross-lingual generalization and emergent cross-modal capabilities enabled by text as a bridge. The work provides a scalable pathway for multimodal and multilingual music understanding and retrieval, along with valuable datasets and benchmarks for future exploration.

Abstract

CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns all major music modalities--including sheet music, performance signals, and audio recordings--with multilingual text in a shared representation space, enabling retrieval across unaligned modalities with text as a bridge. It features a multilingual text encoder adaptable to unseen languages, exhibiting strong cross-lingual generalization. Leveraging retrieval-augmented generation, we curated M4-RAG, a web-scale dataset consisting of 2.31 million music-text pairs. This dataset is enriched with detailed metadata that represents a wide array of global musical traditions. To advance future research, we release WikiMT-X, a benchmark comprising 1,000 triplets of sheet music, audio, and richly varied text descriptions. Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple MIR tasks, significantly surpassing previous strong baselines and demonstrating excellent generalization in multimodal and multilingual music contexts.

Figures (8)

• Figure 1: CLaMP 3 demonstrates robust cross-modal and cross-lingual generalization. Supervised alignment (solid arrows) links paired modalities, while emergent alignment (dashed arrows) bridges unaligned ones. A multilingual text encoder enables retrieval in languages unseen (grayed-out bubbles) during alignment.
• Figure 2: CLaMP 3 uses contrastive learning to align features across modalities. Sheet music and performance signals are segmented into units (bars or MIDI messages) and processed by the symbolic music encoder, while audio is segmented into 5-second clips and processed through the audio feature extractor and audio music encoder. Both symbolic and audio representations are aligned with text representations from the multilingual text encoder.
• Figure 3: Language distribution of original and translated entries in M4-RAG, covering 27 languages.
• Figure 4: Country-wise distribution of music tracks in M4-RAG, spanning 194 countries.
• Figure 5: Genre distribution of the WikiMT-X dataset.