Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals

TL;DR

This work introduces the competition_of_mechanisms framework to study how multiple internal mechanisms in LLMs interact to determine the final prediction, focusing on factual knowledge recall versus counterfactual redefinition. It leverages two interpretable tools—logit_inspection of the residual stream and attention_modification—to locate where and how competition occurs, from macroscopic layer-level dynamics to microscopic attention-head effects. The study finds that late-layer attention blocks and a handful of specialized heads predominantly control the competition, with localized attention entries capable of substantially shifting outcomes toward factual recall. By experimentally perturbing attention and analyzing word-choice similarities, the authors demonstrate both the manipulability of the mechanisms and the sensitivity of predictions to prompt structure, offering insights for interpretability and safety in LLMs.

Abstract

Interpretability research aims to bridge the gap between empirical success and our scientific understanding of the inner workings of large language models (LLMs). However, most existing research focuses on analyzing a single mechanism, such as how models copy or recall factual knowledge. In this work, we propose a formulation of competition of mechanisms, which focuses on the interplay of multiple mechanisms instead of individual mechanisms and traces how one of them becomes dominant in the final prediction. We uncover how and where mechanisms compete within LLMs using two interpretability methods: logit inspection and attention modification. Our findings show traces of the mechanisms and their competition across various model components and reveal attention positions that effectively control the strength of certain mechanisms. Code: https://github.com/francescortu/comp-mech. Data: https://huggingface.co/datasets/francescortu/comp-mech.

Figures (12)

• Figure 1: Top: An example showing that LLMs can fail to recognize the correct mechanism when multiple possible mechanisms exist. Bottom: Our mechanistic inspection of where and how the competition of mechanisms takes place within the LLMs.
• Figure 2: Logits of the factual token $t_{\mathrm{fact}}\xspace$ and counterfactual token $t_{\mathrm{cofa}}\xspace$ across different positions and layers in GPT-2. The logit of $t_{\mathrm{fact}}\xspace$ is higher in the subject position in the initial layers and in the last position of the premise and second sentence in the final layers. The logit of $t_{\mathrm{cofa}}\xspace$ is higher in the attribute position in the first layers and in the last position of the second sentence at the end of the network.
• Figure 3: Contributions of the attention and MLP blocks to the competition of the mechanisms. The attention blocks (left) contribute more to the marginal win of the counterfactual mechanism than the MLP blocks (right).
• Figure 4: Attention pattern for relevant attention heads.
• Figure 5: The factual recall mechanism increases substantially across GPT-2 and Pythia after attention modification.