FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation

TL;DR

FastSAG addresses the slow generation of autoregressive SAG by introducing a diffusion-based, non-autoregressive framework that conditions on vocal signals to directly generate the accompaniment Mel spectrogram. It employs a two-part conditioning pipeline (semantic projection and prior projection) and trains with semantic, prior, and diffusion losses to ensure semantic and rhythmic coherence with the vocals. Empirical results show FastSAG outperforms the AR baseline SingSong on objective metrics and subjective harmony while achieving over 30x real-time speed improvements, enabling real-time SAG. The approach advances practical human-AI collaborative music creation and suggests avenues for more fine-grained, track-level control and potential online deployment in future work.

Abstract

Singing Accompaniment Generation (SAG), which generates instrumental music to accompany input vocals, is crucial to developing human-AI symbiotic art creation systems. The state-of-the-art method, SingSong, utilizes a multi-stage autoregressive (AR) model for SAG, however, this method is extremely slow as it generates semantic and acoustic tokens recursively, and this makes it impossible for real-time applications. In this paper, we aim to develop a Fast SAG method that can create high-quality and coherent accompaniments. A non-AR diffusion-based framework is developed, which by carefully designing the conditions inferred from the vocal signals, generates the Mel spectrogram of the target accompaniment directly. With diffusion and Mel spectrogram modeling, the proposed method significantly simplifies the AR token-based SingSong framework, and largely accelerates the generation. We also design semantic projection, prior projection blocks as well as a set of loss functions, to ensure the generated accompaniment has semantic and rhythm coherence with the vocal signal. By intensive experimental studies, we demonstrate that the proposed method can generate better samples than SingSong, and accelerate the generation by at least 30 times. Audio samples and code are available at https://fastsag.github.io/.

Figures (3)

• Figure 1: Overview of FastSAG. (a) indicates how training data is constructed using a source separation algorithm to acquire vocal-accompaniment pairs. (b) illustrates how to compute conditions based on vocal input. It mainly contains two blocks: semantic projection block and prior projection block. The semantic block is for high-level semantic control and the prior block is for frame-level control. (c) is the stochastic differential equation (SDE) solver, which will take prior computed in (b) as a condition. In the inference process, the generated Mel spectrogram will be converted to an audio waveform through BigvGAN.
• Figure 2: The Prior Projection Block. It contains one resampling module and one encoder module. The resampling module is for reshaping the feature shape, mapping from semantic feature shape to Mel. And encoder module is further for prior generation.
• Figure 3: Illustration of Mel spectrogram. (a) is Mel of the singing voice. (b) is human-composed accompaniment serving as our growth-truth. (c) is predicted prior. (d) is the generated Mel from the diffusion model.