Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT

Sai V R Chereddy

Abstract

The paper explores how video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor: Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.

Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT

Abstract

The paper explores how video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor: Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.
Paper Structure (34 sections, 2 equations, 7 figures, 3 tables)

This paper contains 34 sections, 2 equations, 7 figures, 3 tables.

Table of Contents

  1. 1. Introduction
  2. 2. Related Work
  3. 2.1. Video Vision Transformers
  4. 2.2. Mechanistic Interpretability
  5. 2.3. Eliciting Latent Knowledge
  6. 3. Methodologies
  7. 3.1. Model & Dataset:
  8. Model:
  9. Dataset:
  10. 3.2. Observational Interpretability Analysis:
  11. 3.2.1. Direct Logit Attribution (DLA):
  12. 3.2.2. Token-wise heatmap Visualization:
  13. 3.2.3. CLS Token Attention Visualization:
  14. 3.2.4. Linear Probe Analysis:
  15. 3.3. Signal Identification - Delta Analysis:
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Qualitative visualization of CLS Token attention (Layer 10, Head 8) for the (a) 'Strike' and (b) 'Gutter' videos. This head functions as one of the semantic "outcome-detectors".
  • Figure 2: Contribution of each layer towards the final logits of target class across frames for (a)strike and (b)gutter runs.
  • Figure 3: Token-wise heatmap across frames in "strike run" indicating contribution of each token to the final predicted output class
  • Figure 4: Attention visualisation heatmap for [CLS] token at Layer 9, Head 8.
  • Figure 5: The average L2 norm of the activation delta between the "strike" and "gutter" runs. The sharp rise from layer 5 onwards shows the amplification cascade.
  • ...and 2 more figures