Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

Paweł Pozorski, Jakub Muszyński, Maria Ganzha

TL;DR

Spectrogram-Guided Phonetic Alignment (SGPA), a four-stage pipeline that combines Connectionist Temporal Classification forced alignment with spectral boundary refinement to produce acoustically stable, word-aligned audio segments, is introduced.

Abstract

Explaining the behavior of end-to-end audio language models via Shapley value attribution is intractable under native tokenization: a typical utterance yields over $150$ encoder frames, inflating the coalition space by roughly $10^{42}$ relative to text; individual audio frames lack standalone meaning; and token boundaries that bisect phonetic transitions introduce masking artifacts. We introduce Spectrogram-Guided Phonetic Alignment (SGPA), a four-stage pipeline that combines Connectionist Temporal Classification forced alignment with spectral boundary refinement to produce acoustically stable, word-aligned audio segments. Controlled diagnostics on LFM2-Audio-1.5B with VoiceBench show that SGPA yields a 43$\times$ reduction in model evaluations. Statistical testing confirms that SGPA significantly alters attribution concentration while preserving the global cumulative profile, establishing it as a feasibility-enabling layer for audio explainability.

SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

TL;DR

Spectrogram-Guided Phonetic Alignment (SGPA), a four-stage pipeline that combines Connectionist Temporal Classification forced alignment with spectral boundary refinement to produce acoustically stable, word-aligned audio segments, is introduced.

Abstract

Explaining the behavior of end-to-end audio language models via Shapley value attribution is intractable under native tokenization: a typical utterance yields over encoder frames, inflating the coalition space by roughly relative to text; individual audio frames lack standalone meaning; and token boundaries that bisect phonetic transitions introduce masking artifacts. We introduce Spectrogram-Guided Phonetic Alignment (SGPA), a four-stage pipeline that combines Connectionist Temporal Classification forced alignment with spectral boundary refinement to produce acoustically stable, word-aligned audio segments. Controlled diagnostics on LFM2-Audio-1.5B with VoiceBench show that SGPA yields a 43 reduction in model evaluations. Statistical testing confirms that SGPA significantly alters attribution concentration while preserving the global cumulative profile, establishing it as a feasibility-enabling layer for audio explainability.
Paper Structure (22 sections, 2 equations, 4 figures, 2 tables)

This paper contains 22 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Spectrogram-Guided Phonetic Alignment
  3. Stage 1: Transcript Decomposition
  4. Stage 2: Initial Alignment via CTC
  5. Stage 3: Spectral Boundary Refinement
  6. Stage 4: Word-Level Aggregation
  7. Diagnostic Evaluation
  8. Setup
  9. Model
  10. Dataset
  11. Modes
  12. SV approximation
  13. Infrastructure
  14. Feasibility: Coalition Space Reduction
  15. Attribution Statistics
  16. ...and 7 more sections

Figures (4)

  • Figure 1: SGPA pipeline: transcript decomposition $\to$ CTC alignment $\to$ spectral boundary refinement $\to$ word-level aggregation. The CTC boundary (dashed) is shifted to a spectrally stable region (solid).
  • Figure 2: Distribution of explainable token counts per sample with and without SGPA across modes. SGPA concentrates counts in a narrow regime; native tokenization yields substantially longer and more dispersed sequences.
  • Figure 3: Normalized attribution entropy ($H / \sqrt{n}$) by mode with and without SGPA. SGPA widens the interquartile range, reflecting greater variability in concentration across samples.
  • Figure 4: Position-normalized cumulative SV profile (with derivative) for speech-to-speech mode, illustrating the preserved "early mass" macro-trend under SGPA.