SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation

TL;DR

SPES introduces Spectrogram Perturbation for Explainable Speech-to-text Generation to provide token-level explanations for autoregressive S2T models by accounting for both the input spectrogram $X$ and previously generated tokens $Y^{(k-1)}$. It adopts a perturbation framework using MFPP/SLIC-based spectrogram segmentation and zeroing of token embeddings, estimating impact with KL divergence $KL(oldsymbol{ m ext Psi}( ext{orig}), oldsymbol{ m ext Psi}( ext{perturbed}))$ to produce two saliency maps per token: $S_{y_k}^{X}$ and $S_{y_k}^{Y^{(k-1)}}$, which are then aggregated and normalized. Evaluations on ASR and speech translation show SPES yields faithful (deletion AUC), compact (size AUC), and plausible explanations, with ablations confirming the importance of KL-based impact estimation, morphological segmentation, and multi-scale perturbations. By analyzing time and frequency dimensions and the reliance on previous outputs, SPES demonstrates interpretability aligned with phonetic patterns and linguistic context, advancing explainability for speech generation. The approach provides a practical path toward more transparent S2T systems and supports broader adoption of explainable speech technology in real-world applications.

Abstract

Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations. We address this gap by introducing Spectrogram Perturbation for Explainable Speech-to-text Generation (SPES), a feature attribution technique applicable to sequence generation tasks with autoregressive models. SPES provides explanations for each predicted token based on both the input spectrogram and the previously generated tokens. Extensive evaluation on speech recognition and translation demonstrates that SPES generates explanations that are faithful and plausible to humans.

Figures (11)

• Figure 1: A mel-spectrogram (upper figure) and its segmentation obtained with SLIC (bottom figure).
• Figure 2: Variation of Deletion scores across samples of different lengths.
• Figure 3: $S^{X}$ (top) and $S^{Y^{(k-1)}}$ (bottom) for ASR and ST on the utterance "So what was the next step?". In $S^{Y^{(k-1)}}$, rows show the tokens being explained, and columns the previous output tokens.
• Figure 4: SPES explanation generation time across varying sample lengths and distribution of samples by length.
• Figure 5: Deletion curves for the feature-wise strategy, the Bubble Noise technique, and SPES in the ASR and ST tasks.