Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior

TL;DR

The paper tackles bias in vocal effects style transfer arising from optimisation that relies solely on embedding-distance, by formulating the task as maximum-a-posteriori (MAP) estimation with a Gaussian prior over effect parameters. A differentiable DiffVox-style effects model provides the parameter space, with a likelihood defined from embedding distances via a probabilistic mapping, enabling principled inference-time calibration. On MedleyDB vocal data, the proposed MAP calibration improves objective metrics (e.g., PMSE, MSS, MLDR) over baselines, with AFx-Rep often delivering the best PMSE at suitable prior strength $\alpha$, and subjective listening tests showing a tendency for improved perceived style transfer under limited data. The results demonstrate that incorporating prior knowledge at inference time yields more realistic and controllable vocal effects transfer, and point to promising avenues for stronger, conditional priors and handling variable-dimension effect configurations.

Abstract

Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to an audio track. It optimises the effect parameters to minimise the distance between the style embeddings of the processed audio and the reference. However, this method treats all possible configurations equally and relies solely on the embedding space, which can result in unrealistic configurations or biased outcomes. We address this pitfall by introducing a Gaussian prior derived from the DiffVox vocal preset dataset over the parameter space. The resulting optimisation is equivalent to maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the MedleyDB dataset show significant improvements across metrics compared to baselines, including a blind audio effects estimator, nearest-neighbour approaches, and uncalibrated ST-ITO. The proposed calibration reduces the parameter mean squared error by up to 33% and more closely matches the reference style. Subjective evaluations with 16 participants confirm the superiority of our method in limited data regimes. This work demonstrates how incorporating prior knowledge at inference time enhances audio effects transfer, paving the way for more effective and realistic audio processing systems.

Figures (3)

• Figure 1: Overview of the proposed calibration method. The prior and the likelihood log-probability densities in the parameter space are represented by two concentric ellipses, coloured blue and green, respectively. Darker colours indicate higher density. The coloured arrows indicate the gradients of the log-probability densities. The red star is the optimal parameters $\bm{\theta}^*$ for the vocal effects style transfer.
• Figure 2: The effects chain in DiffVox.
• Figure 3: Violin plot of the average ratings sorted based on the mean. The white dot is the median, and the black thick lines are the interquartile range.