Audio Explanation Synthesis with Generative Foundation Models
Alican Akman, Qiyang Sun, Björn W. Schuller
TL;DR
This work addresses the problem of explaining audio foundation models by moving feature attribution from the input space to a latent embedding space learned by autoencoder-based foundation models. They compute latent attributions with $Z = Encoder(X)$ and $att = Theta(Classifier(Z))$, then synthesize explanations by decoding a modified latent vector $X_Theta = Decoder(Z_Theta)$. The approach is validated on keyword spotting and speech emotion recognition, showing higher fidelity explanations that capture meaningful high-level audio components. The work provides a latent-space explainability framework that enables interpretable, listenable audio explanations and can support model debugging and justification, with potential extensions to advanced audio generative models.
Abstract
The increasing success of audio foundation models across various tasks has led to a growing need for improved interpretability to understand their intricate decision-making processes better. Existing methods primarily focus on explaining these models by attributing importance to elements within the input space based on their influence on the final decision. In this paper, we introduce a novel audio explanation method that capitalises on the generative capacity of audio foundation models. Our method leverages the intrinsic representational power of the embedding space within these models by integrating established feature attribution techniques to identify significant features in this space. The method then generates listenable audio explanations by prioritising the most important features. Through rigorous benchmarking against standard datasets, including keyword spotting and speech emotion recognition, our model demonstrates its efficacy in producing audio explanations.