Streaming Generation for Music Accompaniment

TL;DR

The paper tackles real-time acoustic-to-acoustic music accompaniment by formalizing streaming chunked prediction with design variables $t_f$ (future visibility) and $k$ (output chunk duration). It trains decoder-only Transformer models on the Slakh2100 dataset using residual vector-quantized tokens to study latency, coherence, and throughput across a grid of $(t_f,k)$ settings. Key findings show that small positive $t_f$ improves coherence and rhythm alignment, while larger $k$ increases throughput but reduces the ability to react to the latest input; negative $t_f$ can severely degrade perception, underscoring latency compensation challenges. The work highlights the need for anticipatory and coordinative training objectives for real-time jamming and provides code and pretrained models to enable reproducible research in live audio accompaniment.

Abstract

Music generation models can produce high-fidelity coherent accompaniment given complete audio input, but are limited to editing and loop-based workflows. We study real-time audio-to-audio accompaniment: as a model hears an input audio stream (e.g., a singer singing), it has to also simultaneously generate in real-time a coherent accompanying stream (e.g., a guitar accompaniment). In this work, we propose a model design considering inevitable system delays in practical deployment with two design variables: future visibility $t_f$, the offset between the output playback time and the latest input time used for conditioning, and output chunk duration $k$, the number of frames emitted per call. We train Transformer decoders across a grid of $(t_f,k)$ and show two consistent trade-offs: increasing effective $t_f$ improves coherence by reducing the recency gap, but requires faster inference to stay within the latency budget; increasing $k$ improves throughput but results in degraded accompaniment due to a reduced update rate. Finally, we observe that naive maximum-likelihood streaming training is insufficient for coherent accompaniment where future context is not available, motivating advanced anticipatory and agentic objectives for live jamming.

Figures (8)

• Figure 1: We formalize real-time accompaniment as a streaming, chunked prediction. At each inference step, the model predicts a length-$k$ future chunk given its previous outputs and the input history, with the future visibility $t_f$. Models with $t_f<0$ predicts ahead to compensate for system latency, where the extreme case $t_f=0$ yields synchronous generation. Models with $t_f>0$ generate with input lookahead, which makes aligned playback to the performer infeasible.
• Figure 2: We implement streamed generation models as decoder-only Transformer conditioned on two streams. (Left) Audio of input tracks is mixed and both the mixture and the target stem are tokenized into RVQ codes by a DAC codec. (Middle) We apply the delay pattern to the output stream and prepend instrument token, then align the input and the output with padding to realize $t_f$. (Right) We embed input via frozen DAC codebook and output tokens with a learned embedding, before added and fed to the model, which predicts the next $k$ frames.
• Figure 3: Overall coherence, rhythm alignment, audio quality and subjective preference of generated accompaniment for streaming models with $k=1$ across different $t_f$.
• Figure 4: Overall coherence for different $k>1$ frame and $t_f$.
• Figure 5: Accompaniment performance for streaming models with $k=1$ evaluated across different future visibility $t_f$. Left: performance under different prompt length. Right: performance when conditioning on the paired input compared with conditioning on a random input.