Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpreting token compositionality in LLMs: A robustness analysis

Nura Aljaafari, Danilo S. Carvalho, André Freitas

TL;DR

This paper introduces Constituent-Aware Pooling (CAP), a targeted perturbation method that aggregates token-level activations into word- and phrase-level units to assess where compositional meaning is encoded in Transformer-based LLMs. Using three WordNet-derived tasks (inverse definition modelling, synonyms, and hypernyms), CAP reveals that compositional representations are not reliably localized to any single layer and that information is distributed across depth, with larger models showing greater sensitivity to perturbations. An information-theoretic framework explains these findings as a consequence of delaying aggregation to maximize token-level information throughput, leading to fragmented semantics across layers. Fine-tuning improves resilience but does not fully resolve architectural limitations, suggesting that new training objectives or architectures are needed to realize robust, explicit compositional representations.

Abstract

Understanding the internal mechanisms of large language models (LLMs) is integral to enhancing their reliability, interpretability, and inference processes. We present Constituent-Aware Pooling (CAP), a methodology designed to analyse how LLMs process compositional linguistic structures. Grounded in principles of compositionality, mechanistic interpretability, and information theory, CAP systematically intervenes in model activations through constituent-based pooling at various model levels. Our experiments on inverse definition modelling, hypernym and synonym prediction reveal critical insights into transformers' limitations in handling compositional abstractions. No specific layer integrates tokens into unified semantic representations based on their constituent parts. We observe fragmented information processing, which intensifies with model size, suggesting that larger models struggle more with these interventions and exhibit greater information dispersion. This fragmentation likely stems from transformers' training objectives and architectural design, preventing systematic and cohesive representations. Our findings highlight fundamental limitations in current transformer architectures regarding compositional semantics processing and model interpretability, underscoring the critical need for novel approaches in LLM design to address these challenges.

Interpreting token compositionality in LLMs: A robustness analysis

TL;DR

This paper introduces Constituent-Aware Pooling (CAP), a targeted perturbation method that aggregates token-level activations into word- and phrase-level units to assess where compositional meaning is encoded in Transformer-based LLMs. Using three WordNet-derived tasks (inverse definition modelling, synonyms, and hypernyms), CAP reveals that compositional representations are not reliably localized to any single layer and that information is distributed across depth, with larger models showing greater sensitivity to perturbations. An information-theoretic framework explains these findings as a consequence of delaying aggregation to maximize token-level information throughput, leading to fragmented semantics across layers. Fine-tuning improves resilience but does not fully resolve architectural limitations, suggesting that new training objectives or architectures are needed to realize robust, explicit compositional representations.

Abstract

Understanding the internal mechanisms of large language models (LLMs) is integral to enhancing their reliability, interpretability, and inference processes. We present Constituent-Aware Pooling (CAP), a methodology designed to analyse how LLMs process compositional linguistic structures. Grounded in principles of compositionality, mechanistic interpretability, and information theory, CAP systematically intervenes in model activations through constituent-based pooling at various model levels. Our experiments on inverse definition modelling, hypernym and synonym prediction reveal critical insights into transformers' limitations in handling compositional abstractions. No specific layer integrates tokens into unified semantic representations based on their constituent parts. We observe fragmented information processing, which intensifies with model size, suggesting that larger models struggle more with these interventions and exhibit greater information dispersion. This fragmentation likely stems from transformers' training objectives and architectural design, preventing systematic and cohesive representations. Our findings highlight fundamental limitations in current transformer architectures regarding compositional semantics processing and model interpretability, underscoring the critical need for novel approaches in LLM design to address these challenges.

Paper Structure

This paper contains 32 sections, 12 equations, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Tokenisation and compositionality in LLMs
  3. Assessing compositional aggregation robustness
  4. Empirical analysis
  5. Experimental setup & datasets
  6. Datasets and metrics.
  7. LLMs and evaluated dimensions.
  8. Experimental setup.
  9. Results and discussion
  10. Information Gain & Token Mutual Information
  11. Related work
  12. Conclusion
  13. Compositionality and Localisation
  14. Elaborations on Experimental Setup
  15. Downstream Task Definitions
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Illustration of the CAP process. Constituent segmentation identifies linguistic units (e.g., words or phrases), and CAP pools their activations at layer $m$ using aggregation (e.g., max, mean, sum). This operation reduces sequence length, and the modified activations are propagated to layer $m{+}1$. The results graph shows task accuracy under CAP at different depths.
  • Figure 2: Average grouped accuracy of CAP across different aggregation functions for normalised layer positions (0%-100%) is shown for word-level CAP (TW) and phrasal-level CAP (TP). Sub-figures (a)-(c) illustrate the CAP effect on the original (Org) models, while sub-figures (d)-(f) show its impact on the fine-tuned (FT) models. Fine-tuning consistently improves performance, particularly in the middle to late layers (25%-100%), while early layers (0%-25%) show more variability and lower accuracy across models.
  • Figure 3: Average grouped accuracy of CAP across different aggregation functions for normalised layer positions (0%-100%) is shown for word-level CAP (TW) and phrasal-level CAP (TP). Sub-figures (a)-(c) illustrate the CAP effect on the original (Org) models, while sub-figures (d)-(f) show its impact on the fine-tuned (FT) models. Fine-tuning consistently improves performance, particularly in the middle to late layers (25%-100%), while early layers (0%-25%) show more variability and lower accuracy across models.