Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer

Michele Mancusi, Yurii Halychanskyi, Kin Wai Cheuk, Eloi Moliner, Chieh-Hsin Lai, Stefan Uhlich, Junghyun Koo, Marco A. Martínez-Ramírez, Wei-Hsiang Liao, Giorgio Fabbro, Yuki Mitsufuji

TL;DR

The paper tackles unsupervised musical timbre transfer between monophonic instrument sounds by introducing dual diffusion bridges that operate on instrument-specific latent spaces and map through a Gaussian prior to transfer timbre while preserving melody. By training separate denoisers on source and target domains and solving forward and reverse probability-flow ODEs, the method achieves effective timbre modification without paired data, with a controllable trade-off governed by diffusion noise levels $σ$. The authors provide a distributional cycle-consistency analysis that accounts for discretization and training errors, and validate their approach on the CocoChorales dataset, using EnCodec embeddings and a mix of objective and perceptual metrics alongside a listening test. Compared to VAEGAN and Gaussian Flow Bridges, the proposed framework yields better timbre transfer and melody preservation, and demonstrates practical levers such as pitch-shifting augmentation, chunk-based coupling, and shared latent-space considerations. These results offer a scalable, unsupervised pathway for instrument-to-instrument timbre transfer with robust theoretical grounding and actionable design choices.

Abstract

Music timbre transfer is a challenging task that involves modifying the timbral characteristics of an audio signal while preserving its melodic structure. In this paper, we propose a novel method based on dual diffusion bridges, trained using the CocoChorales Dataset, which consists of unpaired monophonic single-instrument audio data. Each diffusion model is trained on a specific instrument with a Gaussian prior. During inference, a model is designated as the source model to map the input audio to its corresponding Gaussian prior, and another model is designated as the target model to reconstruct the target audio from this Gaussian prior, thereby facilitating timbre transfer. We compare our approach against existing unsupervised timbre transfer models such as VAEGAN and Gaussian Flow Bridges (GFB). Experimental results demonstrate that our method achieves both better Fréchet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB. Additionally, we discover that the noise level from the Gaussian prior, $σ$, can be adjusted to control the degree of melody preservation and amount of timbre transferred.

Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer

TL;DR

The paper tackles unsupervised musical timbre transfer between monophonic instrument sounds by introducing dual diffusion bridges that operate on instrument-specific latent spaces and map through a Gaussian prior to transfer timbre while preserving melody. By training separate denoisers on source and target domains and solving forward and reverse probability-flow ODEs, the method achieves effective timbre modification without paired data, with a controllable trade-off governed by diffusion noise levels . The authors provide a distributional cycle-consistency analysis that accounts for discretization and training errors, and validate their approach on the CocoChorales dataset, using EnCodec embeddings and a mix of objective and perceptual metrics alongside a listening test. Compared to VAEGAN and Gaussian Flow Bridges, the proposed framework yields better timbre transfer and melody preservation, and demonstrates practical levers such as pitch-shifting augmentation, chunk-based coupling, and shared latent-space considerations. These results offer a scalable, unsupervised pathway for instrument-to-instrument timbre transfer with robust theoretical grounding and actionable design choices.

Abstract

Music timbre transfer is a challenging task that involves modifying the timbral characteristics of an audio signal while preserving its melodic structure. In this paper, we propose a novel method based on dual diffusion bridges, trained using the CocoChorales Dataset, which consists of unpaired monophonic single-instrument audio data. Each diffusion model is trained on a specific instrument with a Gaussian prior. During inference, a model is designated as the source model to map the input audio to its corresponding Gaussian prior, and another model is designated as the target model to reconstruct the target audio from this Gaussian prior, thereby facilitating timbre transfer. We compare our approach against existing unsupervised timbre transfer models such as VAEGAN and Gaussian Flow Bridges (GFB). Experimental results demonstrate that our method achieves both better Fréchet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB. Additionally, we discover that the noise level from the Gaussian prior, , can be adjusted to control the degree of melody preservation and amount of timbre transferred.
Paper Structure (18 sections, 1 theorem, 21 equations, 4 tables, 1 algorithm)

This paper contains 18 sections, 1 theorem, 21 equations, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Inference and Training of the Diffusion Bridge
  4. Cycle Consistency
  5. Model Architecture
  6. Experiments
  7. Dataset
  8. Experimental Setup
  9. Pitch-Shifting Augmentation
  10. Chunk-based Minibatch Optimal Transport Coupling
  11. Evaluation
  12. Results
  13. Baseline Comparisons
  14. Effectiveness of Pitch-Shifting Augmentation
  15. Impact of $\sigma_{\text{max}}$ on Timbre Transfer
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\hat{p}^{\text{tgt}}$ denote the density obtained by solving the ODEs in Eq. (eq:ode) numerically via a $\kappa^{\text{th}}$-order RK method, starting from $x^{(s)} \sim p^{\text{src}}$. Let $h := \max_{i} \left|\sigma_{i+1} - \sigma_{i}\right|$ be the discretization timestep. Under certain ass Here, $\epsilon_{\text{DM}}$ represents the training error of diffusion models, and $\mathcal{O}(\c

Theorems & Definitions (2)

  • Theorem 1: Distributional Cycle Consistency
  • proof