Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs

Anand, Umberto Cappellazzo, Stavros Petridis, Maja Pantic

TL;DR

This work analyzes the internal dynamics of audio-visual LLMs, revealing attention sinks at the BOS token and new intermediate sinks that emerge during fine-tuning across ASR, VSR, and AVSR. It shows that massive activations originate in the MLP (notably layer 2) and share identical feature indices across sink tokens, driven by high cosine similarity to the BOS state. The authors propose a lightweight decorrelation loss that reduces BOS-token alignment with other tokens, mitigating both sinks and activations without architectural changes. Empirical results across AVSR, ASR, and VSR demonstrate improved word error rate under high audio-visual downsampling, indicating practical benefits for robust multimodal speech recognition with LLMs.

Abstract

Large language models (LLMs) have recently advanced auditory speech recognition (ASR), visual speech recognition (VSR), and audio-visual speech recognition (AVSR). However, understanding of their internal dynamics under fine-tuning remains limited. In natural language processing, recent work has revealed attention sinks, tokens that attract disproportionately high attention, and associated massive activations in which some features of sink tokens exhibit huge activation in LLMs. In this work, we are the first to study these phenomena in multimodal speech recognition. Through a detailed analysis of audio-visual LLMs, we identify attention sinks and massive activations not only at the BOS token but also at intermediate low-semantic tokens across ASR, VSR, and AVSR. We show that massive activations originate in the MLP layers and correspond to fixed feature indices across all sink tokens. We further show that intermediate sink tokens exhibit high cosine similarity to the BOS token, thereby amplifying attention and activation. Building on these insights, we introduce a simple decorrelation loss that reduces cosine similarity between BOS and other tokens, effectively mitigating intermediate sinks and massive activations. Furthermore, our method improves word error rate (WER) under high audio-visual feature downsampling while remaining stable at lower downsampling rates.

Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs

TL;DR

This work analyzes the internal dynamics of audio-visual LLMs, revealing attention sinks at the BOS token and new intermediate sinks that emerge during fine-tuning across ASR, VSR, and AVSR. It shows that massive activations originate in the MLP (notably layer 2) and share identical feature indices across sink tokens, driven by high cosine similarity to the BOS state. The authors propose a lightweight decorrelation loss that reduces BOS-token alignment with other tokens, mitigating both sinks and activations without architectural changes. Empirical results across AVSR, ASR, and VSR demonstrate improved word error rate under high audio-visual downsampling, indicating practical benefits for robust multimodal speech recognition with LLMs.

Abstract

Large language models (LLMs) have recently advanced auditory speech recognition (ASR), visual speech recognition (VSR), and audio-visual speech recognition (AVSR). However, understanding of their internal dynamics under fine-tuning remains limited. In natural language processing, recent work has revealed attention sinks, tokens that attract disproportionately high attention, and associated massive activations in which some features of sink tokens exhibit huge activation in LLMs. In this work, we are the first to study these phenomena in multimodal speech recognition. Through a detailed analysis of audio-visual LLMs, we identify attention sinks and massive activations not only at the BOS token but also at intermediate low-semantic tokens across ASR, VSR, and AVSR. We show that massive activations originate in the MLP layers and correspond to fixed feature indices across all sink tokens. We further show that intermediate sink tokens exhibit high cosine similarity to the BOS token, thereby amplifying attention and activation. Building on these insights, we introduce a simple decorrelation loss that reduces cosine similarity between BOS and other tokens, effectively mitigating intermediate sinks and massive activations. Furthermore, our method improves word error rate (WER) under high audio-visual feature downsampling while remaining stable at lower downsampling rates.
Paper Structure (13 sections, 11 equations, 4 figures, 2 tables)

This paper contains 13 sections, 11 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Analysis of Audio-Visual LLMs
  4. Attention Sinks
  5. Massive Activations
  6. Cosine Similarity with BOS
  7. Proposed Method
  8. Decorrelation Loss
  9. Final Training Loss
  10. Experimental Results
  11. Decorrelation Loss for Intermediate Attention Sinks
  12. Comparison with Prior Sink Mitigation Methods
  13. Conclusion

Figures (4)

  • Figure 1: (a,b,c) Attention sinks present in BOS and intermediate tokens for different tasks and compression rates (e.g., AVSR task at audio-video compression rates of (16, 5)). (d) Activation magnitudes (z-axis) of the hidden state in Llama-AVSR (16, 5) at layer 5 reveal some features with massive activation in sink tokens.
  • Figure 2: (a) Intermediate attention sinks originate after layer 2 in Llama-AVSR (16,5). (b) Massive activations in Llama-AVSR (16,5) originate from the MLP of Layer 2.
  • Figure 3: (a) Cosine similarity of intermediate sink tokens and non-sink tokens with BOS token across layers of Llama-AVSR (16, 5). (b) Pairwise cosine similarity heatmap of hidden-state embeddings of tokens of Llama-AVSR (16, 5) in layer 5.
  • Figure 4: (a,b) Sinks and massive activations vanish when disaligned from BOS. (c,d) They emerge when aligned with BOS.