I Guess That's Why They Call it the Blues: Causal Analysis for Audio Classifiers

TL;DR

The paper tackles the problem that audio classifiers often rely on spurious, non-musical cues. It introduces freqrex, a causality-based method operating in the frequency domain to identify minimal sufficient, necessary, and complete frequency components, usable in a black-box setting. Empirical results across multiple models and two benchmark datasets show that altering very small subsets of frequencies can flip classifications (e.g., 58% with one frequency, 78% with five; STFT-based changes can be nearly inaudible yet effective). This work provides a principled framework to understand and audit audio classifiers, with implications for robustness and adversarial analysis.

Abstract

It is well-known that audio classifiers often rely on non-musically relevant features and spurious correlations to classify audio. Hence audio classifiers are easy to manipulate or confuse, resulting in wrong classifications. While inducing a misclassification is not hard, until now the set of features that the classifiers rely on was not well understood. In this paper we introduce a new method that uses causal reasoning to discover features of the frequency space that are sufficient and necessary for a given classification. We describe an implementation of this algorithm in the tool FreqReX and provide experimental results on a number of standard benchmark datasets. Our experiments show that causally sufficient and necessary subsets allow us to manipulate the outputs of the models in a variety of ways by changing the input very slightly. Namely, a change to one out of 240,000 frequencies results in a change in classification 58% of the time, and the change can be so small that it is practically inaudible. These results show that causal analysis is useful for understanding the reasoning process of audio classifiers and can be used to successfully manipulate their outputs.

Figures (4)

• Figure 1: An example of freqrex output. (a) shows the original signal, classified as 'blues'. (b) is the responsibility over $\mathcal{F}$. (c) shows the sufficient frequencies (in orange), sufficient and necessary (combined orange and yellow) and finally, in red, the additional frequencies required to achieve the original model softmax score. (d) shows the reconstructed sufficient signal for the blues. This represents the bare minimum that the model will accept of this audio to be 'blues'. (e) is the sufficient and necessary signal. Removing this from (a) gives the class 'hiphop'. Finally (f) adjusts the model score of (e) down towards the original score.
• Figure 2: Average softmax of responsibility across $5$ models on the GTZAN dataset. While not identical, all models across most genres exhibit similar peak distributions, though the relative responsibility of the peaks varies per model. The $x$ axis is Hz.
• Figure 3: Frequency counts for necessary frequencies on GTZAN for correctly classified audio. While each model exhibits different patterns, there is a similarity between the curves of different models within the same genre. The $x$ axis is Hz.
• Figure 4: Two spectrograms, before and after STFT based alteration of the signal. There is a very small (imperceptible) change in amplitude between the two signal and a small noise spike around $1.6s$, yet this is enough to change the classification from 'calm' to 'sad'.

Theorems & Definitions (4)

• Definition 1: Sufficient Subset
• Definition 2: Sufficient responsibility
• Definition 3: Necessary Subset
• Definition 4: Complete Subset