Diffusion Timbre Transfer Via Mutual Information Guided Inpainting

TL;DR

This work tackles timbre transfer for music by reframing it as an inference-time editing problem on a pretrained latent diffusion backbone. It introduces MI-guided, per-channel noise injection and early-step clamping to selectively perturb timbre-dominant latent channels while preserving melodic structure, enabling timbre transfer without retraining. The approach leverages mutual information against instrument labels to identify relevant channels and uses DDIM-based inversion to constrain structural channels, yielding competitive timbre transfer with improved content preservation. The method is lightweight, conditioning-friendly, and applicable to other inference-time edits, offering a practical, scalable tool for creative audio manipulation within pretrained diffusion pipelines.

Abstract

We study timbre transfer as an inference-time editing problem for music audio. Starting from a strong pre-trained latent diffusion model, we introduce a lightweight procedure that requires no additional training: (i) a dimension-wise noise injection that targets latent channels most informative of instrument identity, and (ii) an early-step clamping mechanism that re-imposes the input's melodic and rhythmic structure during reverse diffusion. The method operates directly on audio latents and is compatible with text/audio conditioning (e.g., CLAP). We discuss design choices,analyze trade-offs between timbral change and structural preservation, and show that simple inference-time controls can meaningfully steer pre-trained models for style-transfer use cases.

Figures (2)

• Figure 1: Per-channel mutual information between latent value and instrument class and pitch, and are normalised between [0, 1].
• Figure 2: Cumulative MI for instrument identity at $k=0.5$: left, MI in the top 32 latent channels; right, residual MI in the remaining channels.