Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond

Jessie Richter-Powell, Antonio Torralba, Jonathan Lorraine

TL;DR

This paper addresses the challenge of flexible, prompt-driven audio generation and editing without task-specific datasets. It introduces Audio-SDS, generalizing Score Distillation Sampling to text-conditioned audio diffusion, distilling a pretrained diffusion prior into parametric audio representations. It adds Decoder-SDS, multiscale spectrogram emphasis, and multistep denoising to stabilize and improve fidelity, and demonstrates three core applications—FM synthesis tuning, physically informed impact synthesis, and prompt-driven source separation—showing improved semantic alignment and reconstruction. The results highlight the potential of unified priors to accelerate multimodal audio design and motivate further exploration of distillation-based priors across audio, vision, and beyond.

Abstract

We introduce Audio-SDS, a generalization of Score Distillation Sampling (SDS) to text-conditioned audio diffusion models. While SDS was initially designed for text-to-3D generation using image diffusion, its core idea of distilling a powerful generative prior into a separate parametric representation extends to the audio domain. Leveraging a single pretrained model, Audio-SDS enables a broad range of tasks without requiring specialized datasets. In particular, we demonstrate how Audio-SDS can guide physically informed impact sound simulations, calibrate FM-synthesis parameters, and perform prompt-specified source separation. Our findings illustrate the versatility of distillation-based methods across modalities and establish a robust foundation for future work using generative priors in audio tasks.

Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond

TL;DR

This paper addresses the challenge of flexible, prompt-driven audio generation and editing without task-specific datasets. It introduces Audio-SDS, generalizing Score Distillation Sampling to text-conditioned audio diffusion, distilling a pretrained diffusion prior into parametric audio representations. It adds Decoder-SDS, multiscale spectrogram emphasis, and multistep denoising to stabilize and improve fidelity, and demonstrates three core applications—FM synthesis tuning, physically informed impact synthesis, and prompt-driven source separation—showing improved semantic alignment and reconstruction. The results highlight the potential of unified priors to accelerate multimodal audio design and motivate further exploration of distillation-based priors across audio, vision, and beyond.

Abstract

We introduce Audio-SDS, a generalization of Score Distillation Sampling (SDS) to text-conditioned audio diffusion models. While SDS was initially designed for text-to-3D generation using image diffusion, its core idea of distilling a powerful generative prior into a separate parametric representation extends to the audio domain. Leveraging a single pretrained model, Audio-SDS enables a broad range of tasks without requiring specialized datasets. In particular, we demonstrate how Audio-SDS can guide physically informed impact sound simulations, calibrate FM-synthesis parameters, and perform prompt-specified source separation. Our findings illustrate the versatility of distillation-based methods across modalities and establish a robust foundation for future work using generative priors in audio tasks.
Paper Structure (51 sections, 14 equations, 12 figures, 3 tables)

This paper contains 51 sections, 14 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Contextualizing Diffusion Models
  4. Score Distillation Sampling (SDS)
  5. Related Audio Generative Approaches
  6. Data Driven Audio Tasks
  7. Approach: Audio-SDS and Applications
  8. Using SDS on Audio Diffusion Models
  9. Audio Diffusion Encoder Instabilities
  10. Spectrogram Emphasis for Audio-SDS
  11. Multistep Denoising for Enhanced SDS
  12. Proposed Audio-SDS Applications
  13. FM Synthesizer
  14. Physically Informed Impact Synthesis
  15. Source Separation
  16. ...and 36 more sections

Figures (12)

  • Figure 1: We show an overview of our Audio-SDS framework on the top, our update in the middle, and our different tasks on the bottom. Our framework is further detailed in Sec. \ref{['sec:method']}. A more detailed overview of the update is in Fig. \ref{['fig:audio_sds_overview_detailed']}, highlighting our modifications to the SDS update. Our tasks are outlined further in Sec. \ref{['sec:audio_sds_applications']}, with results in Sec. \ref{['sec:experiments']}.
  • Figure 2: Overview of our Audio-SDS method, marrying the Score Distillation Sampling (SDS) pooledreamfusion with an audio diffusion model in a robust framework for various audio tasks. SDS (see Sec. \ref{['sec:background_sds']}) -- originally developed for text-to-3D generation -- computes an update for rendered data $\mathbf{x}$ in the diffusion models modality (e.g., image or audio), then propagates that update through a differentiable simulation $\mathbf{g}$ to update parameters $\boldsymbol{\theta}$. Intuitively, this nudges the render parameters to make it "more likely" under our conditioning, here the text prompt $\boldsymbol{p}$. Adapting SDS to audio, we propose three modifications shown in red: (a) Decoder-SDS to circumvent differentiating through the encoder (Sec. \ref{['sec:avoiding_instabilities']}), (b) a spectrogram space update for perceptual details (Sec. \ref{['sec:method_spectrogram_updates']}), and (c) multi-step denoising for improved stability Sec. \ref{['sec:method_multi_step_audio_sds']}. We demonstrate three tasks, which, concretely, are choices of the rendering function $\mathbf{g}$shown in blue: (1) tuning an FM synthesizer, (2) tuning physical impact synthesis, and (3) source separation -- see Sec. \ref{['sec:audio_sds_applications']}. Here, we show results for impact synthesis midway during training using the prompt $\boldsymbol{p} \!=\!$ "kick drum, bass, reverb". Optimizable parameters are shown in cyan, and the user-specified prompt and choice of diffusion model are shown in green.
  • Figure 3: FM Synthesis Overview. We visualize the optimizable parameters $\boldsymbol{\theta}$ as dials for an FM user interface, including the FM matrix $\mathbf{A}$ and each operator's ratio, attack, and decay $\{\omega_v, \alpha_v, \delta_v \}_{v=1}^{V}$, where $V = 4$, at the end of optimization for "kick drum, bass, reverb". The optimized output (https://drive.google.com/file/d/182ecHVg32w641PLL3XFoQHKBn8Z_BdJW/view?usp=sharing) provides $+0.13$ CLAP over the initialization (https://drive.google.com/file/d/11JMwLdoQDhML08GbuKjlaCIjamVM1eDa/view?usp=sharing) showing improved prompt alignment, with more results in Fig. \ref{['fig:fm_and_impact_synthesis_overview_qualitative']}.
  • Figure 4: Impact Synthesis Overview. Learned components and final audio for the impact synthesis problem (Sec. \ref{['sec:method_diff_impact']}) with prompt $\boldsymbol{p} =$ "kick drum, bass, reverb". Parameters $\boldsymbol{\theta}$ control object and reverb impulses $\mathbf{I}_{\text{obj}}^{\boldsymbol{\theta}}$, $\mathbf{I}_{\text{reverb}}^{\boldsymbol{\theta}}$ (\ref{['eq:object_impulse']} and \ref{['eq:reverb_impulse']}), whose convolution is the rendered audio $\mathbf{x} = \mathbf{I}_{\text{obj}}^{\boldsymbol{\theta}} \!\!\star \!\mathbf{I}_{\text{reverb}}^{\boldsymbol{\theta}}$. The optimized output (https://drive.google.com/file/d/1ekCge0x2-JSKsw3Rjm1w7F69al81qxkw/view?usp=sharing) provides $+0.1$ CLAP over the initialization (https://drive.google.com/file/d/1j-KS2OFdtptFngSh372canj0NMc1wRtQ/view?usp=sharing) illustrating improved alignment with the prompt.
  • Figure 5: Prompt-Driven Source Separation. Decomposing a single audio mixture of a saxophone and traffic noise into two separate waveforms, each guided by its text prompt. The sum of these separated waveforms reconstructs the original mixture. Audio links: https://drive.google.com/file/d/1rcfa1Er4TGLcXvG7cdPyn0uVrNF4fKaN/view?usp=sharing, https://drive.google.com/file/d/1UNJp4--Ceypc8lxy6EMameyUpCZ_zOoG/view?usp=sharing, https://drive.google.com/file/d/1GW-DEzex9tj73BeF1nnmj4qU8iiKhsyU/view?usp=sharing, https://drive.google.com/file/d/1iF1B7_Mg6zFqMMfI2IfYnApjJBgIi1N5/view?usp=sharing, https://drive.google.com/file/d/197MVw23v2a3fbaK0-1BWecd6P-el5Jev/view?usp=sharing, https://drive.google.com/file/d/1D0tE30sxPDTK3HguzSqE93YyVqW0PQW0/view?usp=sharing.
  • ...and 7 more figures