Investigating the Sensitivity of Pre-trained Audio Embeddings to Common Effects

TL;DR

The paper investigates how pre-trained audio embeddings from OpenL3, PANNs, and CLAP respond to common audio effects by applying parameterized alterations (gain, low-pass, reverberation, bitcrushing) and analyzing deformation in embedding space with canonical correlation analysis (CCA). It reveals a monotone deformation direction with effect strength (high $R^2$ correlations) but a high-dimensional deformation subspace in $\,\mathbb{R}^d$, indicating embeddings do not globally linearize audio effects. Downstream instrument classification shows that projecting out estimated deformation directions yields limited and inconsistent robustness gains, suggesting linear post-processing is insufficient to neutralize sensitivity. The work provides a general framework for sensitivity analysis of foundation-model embeddings to known parameters and highlights the need for more sophisticated debiasing approaches, potentially generalizable to other modalities. $R^2$ values and directional correlations are reported in $R^2$ and related metrics, with the deformation captured in $\oldsymbol{\mathbb{R}}^d$.

Abstract

In recent years, foundation models have significantly advanced data-driven systems across various domains. Yet, their underlying properties, especially when functioning as feature extractors, remain under-explored. In this paper, we investigate the sensitivity to audio effects of audio embeddings extracted from widely-used foundation models, including OpenL3, PANNs, and CLAP. We focus on audio effects as the source of sensitivity due to their prevalent presence in large audio datasets. By applying parameterized audio effects (gain, low-pass filtering, reverberation, and bitcrushing), we analyze the correlation between the deformation trajectories and the effect strength in the embedding space. We propose to quantify the dimensionality and linearizability of the deformation trajectories induced by audio effects using canonical correlation analysis. We find that there exists a direction along which the embeddings move monotonically as the audio effect strength increases, but that the subspace containing the displacements is generally high-dimensional. This shows that pre-trained audio embeddings do not globally linearize the effects. Our empirical results on instrument classification downstream tasks confirm that projecting out the estimated deformation directions cannot generally improve the robustness of pre-trained embeddings to audio effects.

Figures (4)

• Figure 1: UMAP projected visualizations of the deformation trajectories of the embeddings after applying parameterized audio effects. Colormaps represent the effect strength. "$\boldsymbol{\times}$" marks the embeddings of the original audio, while the colored points are those of the effected audio.
• Figure 2: Correlation between the estimated deformation direction and effect strength for collections of audio samples. Cello samples.
• Figure 3: Comparison of singular values of SVD of sample-wise CCA directions and PCA explained variances of original data for cello and three combinations of embedding and audio effect.
• Figure 4: Classification performance comparison (debiasing is equivalent to desensitizing here) in terms of ROC AUC.