Bass Accompaniment Generation via Latent Diffusion

TL;DR

The paper tackles the challenge of generating a bass accompaniment that matches an arbitrary input mix while offering user-controlled timbre. It introduces a two-stage pipeline: an efficient audio autoencoder to produce compact latent representations and a conditional latent diffusion model that maps mix latents to bass latents, augmented with style grounding and diffusion-space classifier-free guidance. The key contributions are the end-to-end trainable autoencoder with spectral losses, a length-general latent diffusion model with Dynamic Positional Bias, and a timbre-control mechanism via style grounding; experiments show the system can produce basslines that align with the input mix and user style. This work advances generative AI for music production by enabling controllable, long-form accompaniment generation that can be grounded to user timbres and integrated into creative workflows.

Abstract

The ability to automatically generate music that appropriately matches an arbitrary input track is a challenging task. We present a novel controllable system for generating single stems to accompany musical mixes of arbitrary length. At the core of our method are audio autoencoders that efficiently compress audio waveform samples into invertible latent representations, and a conditional latent diffusion model that takes as input the latent encoding of a mix and generates the latent encoding of a corresponding stem. To provide control over the timbre of generated samples, we introduce a technique to ground the latent space to a user-provided reference style during diffusion sampling. For further improving audio quality, we adapt classifier-free guidance to avoid distortions at high guidance strengths when generating an unbounded latent space. We train our model on a dataset of pairs of mixes and matching bass stems. Quantitative experiments demonstrate that, given an input mix, the proposed system can generate basslines with user-specified timbres. Our controllable conditional audio generation framework represents a significant step forward in creating generative AI tools to assist musicians in music production.

Figures (3)

• Figure 1: Inference of the system. Noise is concatenated to the latent representation of the conditioning waveform $\mathbf{c_x}$, and $K$ denoising steps are performed to generate $\mathbf{\hat{c}_y}$ which is then decoded to waveform. The representation of a user-specified style sample $\mathbf{c_{style}}$ can be used to ground the generated output to a specific style.
• Figure 2: Left: FAD evaluation of unconditional samples with respect to the number of DDIM inference steps. 64 steps result in the lowest FAD, and we use $K=64$ in all subsequent experiments. Right: FAD evaluation of conditional samples with respect to CFG weights and with varying $\phi$. When higher CFG weights ($>2.5$) are used, the latent rescaling technique results in lower FAD.
• Figure 3: Soft assignments of 25 random input mixes and corresponding generated basslines by a contrastive model (Section \ref{['sec:experiments']}). High diagonal values indicate the generated basslines best match their respective conditional inputs.