Evaluating Sound Similarity Metrics for Differentiable, Iterative Sound-Matching

TL;DR

The paper addresses whether a universally optimal loss function exists for differentiable, iterative sound-matching, arguing that loss effectiveness interacts strongly with the chosen synthesis method. It adopts a controlled setup with four differentiable losses and four differentiable synthesizers, conducting 300 randomized trials per of 16 loss–synthesizer pairs, and evaluates results with automatic metrics (P-Loss, MSS) and manual listening scores. Key findings show that the best loss is highly problem-dependent, with DTW-based envelope matching excelling in amplitude-modulation contexts and SIMSE or spectrogram-based losses performing better for subtractive/noise-filtered synthesis, while manual and automatic evaluations largely agree. The study demonstrates the viability of differentiable sound-matching with simple DSP pipelines and highlights the need for bespoke loss designs and broader synthesis coverage to generalize findings to real-world sound design tasks.

Abstract

Manual sound design with a synthesizer is inherently iterative: an artist compares the synthesized output to a mental target, adjusts parameters, and repeats until satisfied. Iterative sound-matching automates this workflow by continually programming a synthesizer under the guidance of a loss function (or similarity measure) toward a target sound. Prior comparisons of loss functions have typically favored one metric over another, but only within narrow settings: limited synthesis methods, few loss types, often without blind listening tests. This leaves open the question of whether a universally optimal loss exists, or the choice of loss remains a creative decision conditioned on the synthesis method and the sound designer's preference. We propose differentiable iterative sound-matching as the natural extension of the available literature, since it combines the manual approach to sound design with modern advances in machine learning. To analyze the variability of loss function performance across synthesizers, we implemented a mix of four novel and established differentiable loss functions, and paired them with differentiable subtractive, additive, and AM synthesizers. For each of the sixteen synthesizer--loss combinations, we ran 300 randomized sound-matching trials. Performance was measured using parameter differences, spectrogram-distance metrics, and manually assigned listening scores. We observed a moderate level of consistency among the three performance measures. Our post-hoc analysis shows that the loss function performance is highly dependent on the synthesizer. These findings underscore the value of expanding the scope of sound-matching experiments and developing new similarity metrics tailored to specific synthesis techniques rather than pursuing one-size-fits-all solutions.

Figures (5)

• Figure 1: Iterative approach to sound design.
• Figure 2: For each synthesizer program, we assign ranks to the four loss functions (DTW_Envelope, L1_Spec, SIMSE_Spec, JTFS) in three different ways, using MSS, P-Loss, or manually assigned Likert scores.
• Figure 3: Distributions and ranks of the loss functions based on three different performance measures. From left to right, the performance measures are: MSS, P-Loss, and Likert. Higher values indicate better performance. Rank colors are 1234.
• Figure 4: Loss landscapes for BP-Noise with only a high-pass filter parameter. L1_Spec, SIMSE_Spec, and JTFS show clear global minima near the correct parameter, while DTW_Envelope remains flat around the target.
• Figure 5: Loss landscapes for a simplified Noise-AM synthesizer with only amplitude modulation parameter. DTW_Envelope exhibits the smoothest and most informative landscape, explaining its superior performance in amplitude-modulated synthesis.