Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jacobian Scopes: token-level causal attributions in LLMs

Toni J. B. Liu, Baran Zadeoğlu, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls

TL;DR

Jacobian Scopes tackle the problem of attributing LLM next-token predictions to individual prior tokens by exploiting the local linear relationship between inputs and the final hidden state. The authors introduce gradient-based, token-level attributions—Semantic, Fisher, and Temperature Scopes—each defined by a different projection direction and computational profile, enabling focus on specific logits, the full predictive distribution, or model confidence. They formalize the influence scores via the input-output Jacobian and Fisher information, provide efficient computation via backpropagation, and demonstrate applications across instruction, translation, and in-context learning, including bias and time-series extrapolation analyses. The work offers practical, open-source tools for mechanistic interpretability in LLMs, with implications for bias auditing, reasoning elucidation, and forecasting behavior in long-context regimes.

Abstract

Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based, token-level causal attribution methods for interpreting LLM predictions. By analyzing the linearized relations of final hidden state with respect to inputs, Jacobian Scopes quantify how input tokens influence a model's prediction. We introduce three variants - Semantic, Fisher, and Temperature Scopes - which respectively target sensitivity of specific logits, the full predictive distribution, and model confidence (inverse temperature). Through case studies spanning instruction understanding, translation and in-context learning (ICL), we uncover interesting findings, such as when Jacobian Scopes point to implicit political biases. We believe that our proposed methods also shed light on recently debated mechanisms underlying in-context time-series forecasting. Our code and interactive demonstrations are publicly available at https://github.com/AntonioLiu97/JacobianScopes.

Jacobian Scopes: token-level causal attributions in LLMs

TL;DR

Jacobian Scopes tackle the problem of attributing LLM next-token predictions to individual prior tokens by exploiting the local linear relationship between inputs and the final hidden state. The authors introduce gradient-based, token-level attributions—Semantic, Fisher, and Temperature Scopes—each defined by a different projection direction and computational profile, enabling focus on specific logits, the full predictive distribution, or model confidence. They formalize the influence scores via the input-output Jacobian and Fisher information, provide efficient computation via backpropagation, and demonstrate applications across instruction, translation, and in-context learning, including bias and time-series extrapolation analyses. The work offers practical, open-source tools for mechanistic interpretability in LLMs, with implications for bias auditing, reasoning elucidation, and forecasting behavior in long-context regimes.

Abstract

Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based, token-level causal attribution methods for interpreting LLM predictions. By analyzing the linearized relations of final hidden state with respect to inputs, Jacobian Scopes quantify how input tokens influence a model's prediction. We introduce three variants - Semantic, Fisher, and Temperature Scopes - which respectively target sensitivity of specific logits, the full predictive distribution, and model confidence (inverse temperature). Through case studies spanning instruction understanding, translation and in-context learning (ICL), we uncover interesting findings, such as when Jacobian Scopes point to implicit political biases. We believe that our proposed methods also shed light on recently debated mechanisms underlying in-context time-series forecasting. Our code and interactive demonstrations are publicly available at https://github.com/AntonioLiu97/JacobianScopes.
Paper Structure (16 sections, 31 equations, 15 figures, 1 table)

This paper contains 16 sections, 31 equations, 15 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Applications
  4. Semantic Scope
  5. Fisher Scope
  6. Temperature Scope
  7. Discussion and conclusion
  8. Appendix
  9. Additional experiments on longer contexts
  10. Dynamical systems: definitions and additional experiments
  11. Semantic vs. Temperature Scopes
  12. Fisher vs. Temperature Scopes
  13. Path-Integrated Semantic Scope
  14. Geometric interpretation of influence under ep -norm perturbations
  15. Second-order KL expansion and Fisher pullback
  16. ...and 1 more sections

Figures (15)

  • Figure 1: Semantic Scope visualizes how LLaMA-3.2 understands system-prompt-style instructions. The logit assigned to the token "truthful" is most prominently attributed to the input token "deceive", and "nice" to "argue".
  • Figure 2: Semantic Scope reveals the potential for political bias that may be present in models such as LLaMA-3.2. The prediction for the subject being a "liberal" is attributed to the input token "Columbia", while "conservative" to the token pairs for "the South".
  • Figure 3: Fisher Scope reveals that translated words, highlighted in red, are primarily influenced by their corresponding words in the source language shaded in blue.
  • Figure 4: LLaMA-3.2 extrapolates a chaotic time-series via pattern matching. Temperature Scope reveals that when predicting future values in a partially observed Lorenz system, LLaMA-3.2 attends to regions in the input history that exhibit peaked patterns similar to those near the cutoff.
  • Figure 5: Temperature Scope shows diminished influence of early context as LLaMA-3.2 extrapolates a Brownian motion.
  • ...and 10 more figures