Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding

Jingyue Huang, Zachary Novack, Phillip Long, Yupeng Hou, Ke Chen, Taylor Berg-Kirkpatrick, Julian McAuley

TL;DR

MuseTok introduces a discrete symbolic-music representation by applying residual quantized variational autoencoders to bar-level segments, learning codebooks with depth $D$ and size $K$ and mapping symbolic sequences $X$ to code sequences that are decoded by a Transformer. It demonstrates high-fidelity reconstruction and strong semantic understanding, achieving superior accuracy on chord recognition and emotion classification while maintaining competitive generation quality. The approach supports a two-stage generation pipeline where a code sequence of length $T_c$ is first produced and then decoded into REMI+ events, enabling long-context generation and interpretable musical concepts such as rhythm, texture, and intervals. The work releases code and checkpoints and points to adaptive tokenization as future work.

Abstract

Discrete representation learning has shown promising results across various domains, including generation and understanding in image, speech and language. Inspired by these advances, we propose MuseTok, a tokenization method for symbolic music, and investigate its effectiveness in both music generation and understanding tasks. MuseTok employs the residual vector quantized-variational autoencoder (RQ-VAE) on bar-wise music segments within a Transformer-based encoder-decoder framework, producing music codes that achieve high-fidelity music reconstruction and accurate understanding of music theory. For comprehensive evaluation, we apply MuseTok to music generation and semantic understanding tasks, including melody extraction, chord recognition, and emotion recognition. Models incorporating MuseTok outperform previous representation learning baselines in semantic understanding while maintaining comparable performance in content generation. Furthermore, qualitative analyses on MuseTok codes, using ground-truth categories and synthetic datasets, reveal that MuseTok effectively captures underlying musical concepts from large music collections.

MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding

TL;DR

MuseTok introduces a discrete symbolic-music representation by applying residual quantized variational autoencoders to bar-level segments, learning codebooks with depth and size and mapping symbolic sequences to code sequences that are decoded by a Transformer. It demonstrates high-fidelity reconstruction and strong semantic understanding, achieving superior accuracy on chord recognition and emotion classification while maintaining competitive generation quality. The approach supports a two-stage generation pipeline where a code sequence of length is first produced and then decoded into REMI+ events, enabling long-context generation and interpretable musical concepts such as rhythm, texture, and intervals. The work releases code and checkpoints and points to adaptive tokenization as future work.

Abstract

Discrete representation learning has shown promising results across various domains, including generation and understanding in image, speech and language. Inspired by these advances, we propose MuseTok, a tokenization method for symbolic music, and investigate its effectiveness in both music generation and understanding tasks. MuseTok employs the residual vector quantized-variational autoencoder (RQ-VAE) on bar-wise music segments within a Transformer-based encoder-decoder framework, producing music codes that achieve high-fidelity music reconstruction and accurate understanding of music theory. For comprehensive evaluation, we apply MuseTok to music generation and semantic understanding tasks, including melody extraction, chord recognition, and emotion recognition. Models incorporating MuseTok outperform previous representation learning baselines in semantic understanding while maintaining comparable performance in content generation. Furthermore, qualitative analyses on MuseTok codes, using ground-truth categories and synthetic datasets, reveal that MuseTok effectively captures underlying musical concepts from large music collections.
Paper Structure (13 sections, 4 equations, 3 figures, 3 tables)

This paper contains 13 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Music Tokenization
  5. Music Generation
  6. Music Understanding
  7. Experiments
  8. Datasets and Pre-processing
  9. Music Tokenization
  10. Music Generation
  11. Music Understanding
  12. How MuseTok Learns Music?
  13. Conclusion

Figures (3)

  • Figure 1: Overview of MuseTok (left) and its downstream generation (middle) and understanding (right) tasks.
  • Figure 2: Top-50 used codes across three texture groups, or across six time signatures in the first ($d$=1) and last ($d$=8) codebook.
  • Figure 3: The cosine similarity of code embeddings ($y$-axis) between original and transposed samples by pitch shifts ($x$-axis) across codebooks. Different colored lines denote different codebooks.