StemGen: A music generation model that listens

TL;DR

StemGen introduces a context-aware, end-to-end music generation framework that listens to musical context and responds with contextually coherent stems. It employs a non-autoregressive transformer operating on tokenized audio via encodec RVQ, fusing multiple audio channels and conditioning signals into a single sequence element. Two novel sampling innovations—causal-bias iterative decoding and multi-source classifier-free-guidance—improve musical coherence and alignment to context. Trained on Slakh and a large internal dataset, StemGen achieves audio quality on par with state-of-the-art text-conditioned models and demonstrates strong context coherence through objective MIR-based metrics and MOS evaluations. The work advances interactive, context-driven music generation with practical implications for production workflows and creative exploration.

Abstract

End-to-end generation of musical audio using deep learning techniques has seen an explosion of activity recently. However, most models concentrate on generating fully mixed music in response to abstract conditioning information. In this work, we present an alternative paradigm for producing music generation models that can listen and respond to musical context. We describe how such a model can be constructed using a non-autoregressive, transformer-based model architecture and present a number of novel architectural and sampling improvements. We train the described architecture on both an open-source and a proprietary dataset. We evaluate the produced models using standard quality metrics and a new approach based on music information retrieval descriptors. The resulting model reaches the audio quality of state-of-the-art text-conditioned models, as well as exhibiting strong musical coherence with its context.

Figures (5)

• Figure 1: Schematic diagram of the StemGen training paradigm.
• Figure 2: Schematic showing overall architecture of the StemGen model during training.
• Figure 3: Schematic showing how the $Q$ RVQ levels of both the context-mix and the target stem are converted into continuous embeddings using codebooks $E_{1 \hdots Q}$, and then combined with each other along with conditioning information.
• Figure 4: Example masking pattern when training to generate the 2nd level of a $Q$-level audio tokenizer. $q_{1\hdots Q}$ denote the RVQ levels, and $s_{0\hdots n}$ the sequence elements (equivalent to time steps in this case).
• Figure 5: Example of iterative decoding for sequence of a single token level over 8 iterations, with and without causal-bias. Black denotes which sequence elements have been sampled.