Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mix2Morph: Learning Sound Morphing from Noisy Mixes

Annie Chu, Hugo Flores García, Oriol Nieto, Justin Salamon, Bryan Pardo, Prem Seetharaman

TL;DR

Mix2Morph addresses the challenge of generating perceptually coherent sound morphs without a dedicated morph dataset by finetuning a pretrained text-to-audio diffusion model on noisy surrogate mixes using a no-waste strategy at high diffusion timesteps ($t \approx 1$). The method combines temporal RMS anchoring and spectral interpolation to create surrogate morphs and employs four augmentation modes to diversify targets, with training focused on $t \in [0.5,1]$ to balance primary identity and infused timbres. The authors introduce evaluation metrics including Latent Compressibility Score (LCS) and a morph-semantic framework (Correspondence, Intermediateness, Directionality) plus FAD, and validate via both objective metrics and a listening study, demonstrating that Mix2Morph outperforms baselines in morph quality and morph rate. The work advances controllable, concept-driven sound design tools by showing that surrogate data and high-timestep training can yield robust infusions across diverse categories, with strong practical implications for audio synthesis and design workflows.

Abstract

We introduce Mix2Morph, a text-to-audio diffusion model fine-tuned to perform sound morphing without a dedicated dataset of morphs. By finetuning on noisy surrogate mixes at higher diffusion timesteps, Mix2Morph yields stable, perceptually coherent morphs that convincingly integrate qualities of both sources. We specifically target sound infusions, a practically and perceptually motivated subclass of morphing in which one sound acts as the dominant primary source, providing overall temporal and structural behavior, while a secondary sound is infused throughout, enriching its timbral and textural qualities. Objective evaluations and listening tests show that Mix2Morph outperforms prior baselines and produces high-quality sound infusions across diverse categories, representing a step toward more controllable and concept-driven tools for sound design. Sound examples are available at https://anniejchu.github.io/mix2morph .

Mix2Morph: Learning Sound Morphing from Noisy Mixes

TL;DR

Mix2Morph addresses the challenge of generating perceptually coherent sound morphs without a dedicated morph dataset by finetuning a pretrained text-to-audio diffusion model on noisy surrogate mixes using a no-waste strategy at high diffusion timesteps (). The method combines temporal RMS anchoring and spectral interpolation to create surrogate morphs and employs four augmentation modes to diversify targets, with training focused on to balance primary identity and infused timbres. The authors introduce evaluation metrics including Latent Compressibility Score (LCS) and a morph-semantic framework (Correspondence, Intermediateness, Directionality) plus FAD, and validate via both objective metrics and a listening study, demonstrating that Mix2Morph outperforms baselines in morph quality and morph rate. The work advances controllable, concept-driven sound design tools by showing that surrogate data and high-timestep training can yield robust infusions across diverse categories, with strong practical implications for audio synthesis and design workflows.

Abstract

We introduce Mix2Morph, a text-to-audio diffusion model fine-tuned to perform sound morphing without a dedicated dataset of morphs. By finetuning on noisy surrogate mixes at higher diffusion timesteps, Mix2Morph yields stable, perceptually coherent morphs that convincingly integrate qualities of both sources. We specifically target sound infusions, a practically and perceptually motivated subclass of morphing in which one sound acts as the dominant primary source, providing overall temporal and structural behavior, while a secondary sound is infused throughout, enriching its timbral and textural qualities. Objective evaluations and listening tests show that Mix2Morph outperforms prior baselines and produces high-quality sound infusions across diverse categories, representing a step toward more controllable and concept-driven tools for sound design. Sound examples are available at https://anniejchu.github.io/mix2morph .
Paper Structure (14 sections, 2 figures, 1 table)

This paper contains 14 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Augmentation Techniques
  4. Augmentation Modes
  5. Morphing Evaluation Metrics
  6. Experiments & Results
  7. Curating Concept Pairs for Evaluation
  8. Model Implementation Details
  9. Effect of Training at Higher Timesteps
  10. Effect of Augmentation Modes
  11. Baselines for Comparison
  12. Quantitative Results
  13. Subjective Evaluation
  14. Conclusion

Figures (2)

  • Figure 1: Mix2Morph pipeline for creating noisy surrogate morphs via augmented additive mixing of sound pairs, used as high-timestep training targets in our diffusion-based TTA model. This example (as well as others) can be listened to at https://anniejchu.github.io/mix2morph/
  • Figure 2: Listener Study Results: Mix2Morph achieves the highest morph rate, with MOS skewed toward higher values.