Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models

Alhassan Abdelhalim, Janick Edinger, Sören Laue, Michaela Regneri

TL;DR

Large language models are increasingly embedded in pervasive and edge computing systems, creating a need for reliable interpretability methods. The study reuses two standard explainability pipelines—attention-based relational analysis and embedding-based property inference—to detect linguistic abstraction in transformer modules. Through controlled perturbations and ablations, the authors show that token identity dissolves in deep layers and that embedding-property scores can be artifacts of geometry and sparsity rather than actual semantic encoding. The work highlights risks of trusting popular explanations in real-world deployments and argues for mechanism-aware interpretability and explicit assumption testing.

Abstract

Large Language Models (LLMs) are becoming increasingly popular in pervasive computing due to their versatility and strong performance. However, despite their ubiquitous use, the exact mechanisms underlying their outstanding performance remain unclear. Different methods for LLM explainability exist, and many are, as a method, not fully understood themselves. We started with the question of how linguistic abstraction emerges in LLMs, aiming to detect it across different LLM modules (attention heads and input embeddings). For this, we used methods well-established in the literature: (1) probing for token-level relational structures, and (2) feature-mapping using embeddings as carriers of human-interpretable properties. Both attempts failed for different methodological reasons: Attention-based explanations collapsed once we tested the core assumption that later-layer representations still correspond to tokens. Property-inference methods applied to embeddings also failed because their high predictive scores were driven by methodological artifacts and dataset structure rather than meaningful semantic knowledge. These failures matter because both techniques are widely treated as evidence for what LLMs supposedly understand, yet our results show such conclusions are unwarranted. These limitations are particularly relevant in pervasive and distributed computing settings where LLMs are deployed as system components and interpretability methods are relied upon for debugging, compression, and explaining models.

LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models

TL;DR

Large language models are increasingly embedded in pervasive and edge computing systems, creating a need for reliable interpretability methods. The study reuses two standard explainability pipelines—attention-based relational analysis and embedding-based property inference—to detect linguistic abstraction in transformer modules. Through controlled perturbations and ablations, the authors show that token identity dissolves in deep layers and that embedding-property scores can be artifacts of geometry and sparsity rather than actual semantic encoding. The work highlights risks of trusting popular explanations in real-world deployments and argues for mechanism-aware interpretability and explicit assumption testing.

Abstract

Large Language Models (LLMs) are becoming increasingly popular in pervasive computing due to their versatility and strong performance. However, despite their ubiquitous use, the exact mechanisms underlying their outstanding performance remain unclear. Different methods for LLM explainability exist, and many are, as a method, not fully understood themselves. We started with the question of how linguistic abstraction emerges in LLMs, aiming to detect it across different LLM modules (attention heads and input embeddings). For this, we used methods well-established in the literature: (1) probing for token-level relational structures, and (2) feature-mapping using embeddings as carriers of human-interpretable properties. Both attempts failed for different methodological reasons: Attention-based explanations collapsed once we tested the core assumption that later-layer representations still correspond to tokens. Property-inference methods applied to embeddings also failed because their high predictive scores were driven by methodological artifacts and dataset structure rather than meaningful semantic knowledge. These failures matter because both techniques are widely treated as evidence for what LLMs supposedly understand, yet our results show such conclusions are unwarranted. These limitations are particularly relevant in pervasive and distributed computing settings where LLMs are deployed as system components and interpretability methods are relied upon for debugging, compression, and explaining models.
Paper Structure (25 sections, 2 figures, 3 tables)

This paper contains 25 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Attention-Based Explanations
  4. Embedding-Based Property Inference
  5. Assumptions Under Test
  6. Methodology
  7. Attention-Based Explainability
  8. Intended Hypothesis and Assumptions
  9. Models and Data
  10. Standard Pipeline and Visual Analysis
  11. Property Inference From Embeddings
  12. Intended Hypothesis and Assumptions
  13. Datasets and Feature Norms
  14. Embedding Extraction
  15. Mapping Methods
  16. ...and 10 more sections

Figures (2)

  • Figure 1: An exemplary visualization of attention head activations in BERT devlin-etal-2019-bert, created with BertViz 1361981470578490112. We argue that the assumption that words have the same embedding positions across layers is not defensible in the inner layers.
  • Figure 2: Pipeline for mapping Embeddings. First the original feature norms are used, then random or nonsensical ones.