Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Learning from Synthetic Audio Doppelgängers

Manuel Cherep, Nikhil Singh

TL;DR

The paper tackles data scarcity in audio representation learning by introducing audio doppelgängers: synthetic positives generated through randomly perturbed synthesizer parameters controlled by a single hyperparameter $\delta$. This approach enables causal variations in timbre, pitch, and envelopes, forming informative pairs for contrastive learning without needing real recordings. Across eight downstream tasks, models trained on synthetic data match or exceed real-data baselines, with $\delta=0.25$ often delivering the strongest performance while maintaining light computational requirements and no on-disk data storage. The authors analyze how synthetic data differ from real data in spectral features and causal uncertainty, and they propose synthetic data as a practical complement to existing augmentation strategies for scalable audio understanding.

Abstract

Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelgängers-synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.

Contrastive Learning from Synthetic Audio Doppelgängers

TL;DR

The paper tackles data scarcity in audio representation learning by introducing audio doppelgängers: synthetic positives generated through randomly perturbed synthesizer parameters controlled by a single hyperparameter . This approach enables causal variations in timbre, pitch, and envelopes, forming informative pairs for contrastive learning without needing real recordings. Across eight downstream tasks, models trained on synthetic data match or exceed real-data baselines, with often delivering the strongest performance while maintaining light computational requirements and no on-disk data storage. The authors analyze how synthetic data differ from real data in spectral features and causal uncertainty, and they propose synthetic data as a practical complement to existing augmentation strategies for scalable audio understanding.

Abstract

Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelgängers-synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.
Paper Structure (28 sections, 6 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning from Synthetic Data
  4. Contrastive Learning
  5. Sound Synthesis
  6. Methods
  7. Data Generation
  8. Synthesis perturbation factor ($\delta$)
  9. Real Data
  10. Preprocessing, Data Augmentations, and Audio Encoder
  11. Contrastive Learning
  12. Evaluation Tasks
  13. Results
  14. Benchmark Results
  15. Characterizing the Data Distribution
  16. ...and 13 more sections

Figures (6)

  • Figure 1: (Left) Standard data augmentation techniques for contrastive learning applied to audio spectrograms (Right)Audio Doppelgängers, our approach synthesizing sounds that are controllably different using perturbed synthesis parameters, shown for different factors $\delta$. These sounds can vary in causally controllable ways beyond what data augmentations can achieve.
  • Figure 2: (A: Top) Average CLAP wu2023large embedding cosine similarity between positive pairs for different architectures and different values of $\delta$. (B: Bottom) PCA of CLAP embeddings for sounds generated with the Voice architecture, with line segments showing distances between paired examples. Red and blue points are paired positive instances. Across both plots, as $\delta$ increases, the positive pairs systematically become more perceptually dissimilar (via the CLAP embedding proxy).
  • Figure 3: Comparisons of synthetic and real sound data (VGGSound chen2020vggsound) on (A: Top) spectral features and (B: Bottom) causal uncertainty. Spectral features of synthetic sounds partially replicate real sounds, but exhibit differences in complexity and flux. Synthetic sounds are also more causally ambiguous, indicating a distribution shift. Using dense mixtures of real sounds partially closes these gaps, suggesting the synthetic sounds are different in part due to their density of auditory information.
  • Figure 4: Scores with the Voice architecture and different values of $\delta$ for evaluation tasks in \ref{['tab:results']} with and without augmentations. $\delta=0.25$ tends to give the best results overall.
  • Figure 5: Scores with a fixed $\delta = 0.25$ and different synthesizer architectures for a suite of tasks including (from left to right) UrbanSound8k salamon2014dataset, ESC-50 piczak2015esc, LibriCount stoter2018libricount, CREMA-D cao2014crema, VIVAE holz2022variably, NSynth Pitch 5h engel2017neural, FSD50k fonseca2021fsd50k, and Vocal Imitation kim2018vocal
  • ...and 1 more figures