Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring Uncertainty in Transformer Circuits with Effective Information Consistency

Anatoly A. Krasnovsky

TL;DR

The paper tackles the challenge of assessing when a Transformer Circuit operates coherently by introducing EICS, a white-box, single-pass score that combines a normalized sheaf-based inconsistency measure with a Gaussian effective-information proxy for causal emergence.By modeling transformer subgraphs as cellular sheaves on a graph and deriving a log-det EI proxy from local Jacobians, the authors provide a dimensionless, computable metric that highlights both integration of information and internal disagreement.A node-seeded JVP approach enables efficient, exact single-pass computation, with fast and exact modes for the EI proxy and practical guidance on interpretation, overhead, and potential normalization strategies.The framework is complemented by a theoretical discussion of stability and bounds via the sheaf Laplacian and a detailed validation protocol, including toy sanity checks and plans for empirical evaluation on factual QA tasks.

Abstract

Mechanistic interpretability has identified functional subgraphs within large language models (LLMs), known as Transformer Circuits (TCs), that appear to implement specific algorithms. Yet we lack a formal, single-pass way to quantify when an active circuit is behaving coherently and thus likely trustworthy. Building on prior systems-theoretic proposals, we specialize a sheaf/cohomology and causal emergence perspective to TCs and introduce the Effective-Information Consistency Score (EICS). EICS combines (i) a normalized sheaf inconsistency computed from local Jacobians and activations, with (ii) a Gaussian EI proxy for circuit-level causal emergence derived from the same forward state. The construction is white-box, single-pass, and makes units explicit so that the score is dimensionless. We further provide practical guidance on score interpretation, computational overhead (with fast and exact modes), and a toy sanity-check analysis. Empirical validation on LLM tasks is deferred.

Measuring Uncertainty in Transformer Circuits with Effective Information Consistency

TL;DR

The paper tackles the challenge of assessing when a Transformer Circuit operates coherently by introducing EICS, a white-box, single-pass score that combines a normalized sheaf-based inconsistency measure with a Gaussian effective-information proxy for causal emergence.By modeling transformer subgraphs as cellular sheaves on a graph and deriving a log-det EI proxy from local Jacobians, the authors provide a dimensionless, computable metric that highlights both integration of information and internal disagreement.A node-seeded JVP approach enables efficient, exact single-pass computation, with fast and exact modes for the EI proxy and practical guidance on interpretation, overhead, and potential normalization strategies.The framework is complemented by a theoretical discussion of stability and bounds via the sheaf Laplacian and a detailed validation protocol, including toy sanity checks and plans for empirical evaluation on factual QA tasks.

Abstract

Mechanistic interpretability has identified functional subgraphs within large language models (LLMs), known as Transformer Circuits (TCs), that appear to implement specific algorithms. Yet we lack a formal, single-pass way to quantify when an active circuit is behaving coherently and thus likely trustworthy. Building on prior systems-theoretic proposals, we specialize a sheaf/cohomology and causal emergence perspective to TCs and introduce the Effective-Information Consistency Score (EICS). EICS combines (i) a normalized sheaf inconsistency computed from local Jacobians and activations, with (ii) a Gaussian EI proxy for circuit-level causal emergence derived from the same forward state. The construction is white-box, single-pass, and makes units explicit so that the score is dimensionless. We further provide practical guidance on score interpretation, computational overhead (with fast and exact modes), and a toy sanity-check analysis. Empirical validation on LLM tasks is deferred.

Paper Structure

This paper contains 23 sections, 2 theorems, 11 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Transformer circuits
  5. Cellular sheaves on graphs and a computable inconsistency
  6. Node-seeded JVP computation (efficiency).
  7. A single-pass, Gaussian EI proxy
  8. Gaussian EI Proxy — Derivation and Implementation Notes
  9. Setup.
  10. Mutual information.
  11. Proxy definition.
  12. Invariance, sensitivity, and small-$\alpha$ approximation.
  13. Computation.
  14. Method: EICS
  15. Why this fixes earlier issues.
  16. ...and 8 more sections

Key Result

proposition 1

Under Assumption assump:lin, both $C_{\mathrm{sh}}(G_M,a)$ and $\widetilde{\Delta \mathrm{EI}}_G(G_M)$ are deterministic functions of a single forward pass and its Jacobian-vector products. Consequently, $\mathrm{EICS}$ is $O(1)$ in the number of forward passes.

Figures (1)

  • Figure 1: Toy sanity-check on a 6-node circuit with two parallel branches. As node-noise $\tau$ increases, the sheaf inconsistency $C_{\mathrm{sh}}$ rises (so $1/(1+C_{\mathrm{sh}})$ falls). We also reduce cross-branch alignment with $\tau$ (edge decoherence), causing the emergence proxy $\widetilde{\Delta \mathrm{EI}}_G$ and the overall EICS to decrease. Curves show means over seeds (no error bands for clarity). Definitions follow Eqs. \ref{['eq:Csh']}, \ref{['eq:EIproxy']}, and \ref{['eq:eics']}.

Theorems & Definitions (2)

  • proposition 1: Single-pass computability
  • proposition 2: Stability to small off-circuit perturbations (bound)