An Independence-promoting Loss for Music Generation with Language Models

TL;DR

This work tackles the inefficiency of jointly modeling multiple codebooks in text-conditioned music generation by introducing an independence-promoting loss based on the maximum mean discrepancy in a universal RKHS. The loss encourages independence between codebooks in the auto-encoder tokenizer, enabling effective factorized modeling with a delay-based decoding strategy while maintaining, or improving, audio quality and generation speed. Empirical results on MusicCaps show improved objective metrics (e.g., FAD) and subjective quality (MOS) compared with strong baselines, and ablations confirm that aligning the MMD optimization with the decoding strategy yields the best performance. The approach is codec-agnostic, scalable, and can be applied to other multi-stream codecs, offering a practical path to faster, high-quality music generation from language models.

Abstract

Music generation schemes using language modeling rely on a vocabulary of audio tokens, generally provided as codes in a discrete latent space learnt by an auto-encoder. Multi-stage quantizers are often employed to produce these tokens, therefore the decoding strategy used for token prediction must be adapted to account for multiple codebooks: either it should model the joint distribution over all codebooks, or fit the product of the codebook marginal distributions. Modelling the joint distribution requires a costly increase in the number of auto-regressive steps, while fitting the product of the marginals yields an inexact model unless the codebooks are mutually independent. In this work, we introduce an independence-promoting loss to regularize the auto-encoder used as the tokenizer in language models for music generation. The proposed loss is a proxy for mutual information based on the maximum mean discrepancy principle, applied in reproducible kernel Hilbert spaces. Our criterion is simple to implement and train, and it is generalizable to other multi-stream codecs. We show that it reduces the statistical dependence between codebooks during auto-encoding. This leads to an increase in the generated music quality when modelling the product of the marginal distributions, while generating audio much faster than the joint distribution model.

Figures (6)

• Figure 1: MusicGen framework. The EnCodec audio auto-encoder (top) encodes the waveform and audio tokens (middle) are obtained by discretizing the encoded audio with the RVQ multi-stage quantizer. The resulting audio tokens are then passed along text embeddings (bottom-left) to a Transformer-style language model with $L$ layers (bottom-right). The language model auto-regressively estimates the next token (right) according to the "delay" decoding strategy kharitonov2022textfree. At the time step $t=7$, our proposed method MusicGen-MMD regularizes the EnCodec bottleneck with the loss $\mathcal{L}_\mathrm{inde}$, thereby promoting independence between the delayed codes $\{ Z_1^7, Z_2^6, Z_3^5, Z_4^4 \}$ produced by RVQ.
• Figure 2: MMD, total correlation of EnCodec codes and MSSpec loss computed on our internal set. MSSpec is the combination of $L1$ and $L2$ losses on the multi-resolution mel-spectrogram, used for reconstruction in EnCodec. The horizontal axis shows the weighting factor used for the MMD loss $\mathcal{L}_\mathrm{inde}$. The total correlation $\mathcal{I}$ is computed on the whole 250k-samples training set for minimal bias in the histogram approximation. It is computed between two codebooks taken at random, averaged over five codebook couples, and expressed as a ratio to the entropy of the joint distribution (in %).
• Figure 3: MMD, Mutual Information of RVQGAN kumar2024highfidelity codes and MSSpec loss computed on our internal set. MSSpec is the combination of $L1$ and $L2$ losses on the multi-resolution mel-spectrogram, used for reconstruction in EnCodec. The horizontal axis shows the weighting factor used for the MMD loss $\mathcal{L}_\mathrm{inde}$. The total correlation $\mathcal{I}$ is computed on the whole 250k-samples training set for minimal bias in the histogram approximation. It is computed between two codebooks taken at random, averaged over five codebook couples, and expressed as a ratio to the entropy of the joint distribution (in %). We removed the data point for the MMD weight of $10^4$ as the experiment diverged.
• Figure 4: Mutual information between individual codebooks (on the horizontal axis) and all other codebooks, for difference codecs on FMA-Pop gui2024adapting.
• Figure 5: Mutual information between individual codebooks (on the horizontal axis) and all other codebooks, for difference codecs on LibriSpeech.