Patterning: The Dual of Interpretability

TL;DR

This work defines patterning as the reverse problem of interpretability: given a desired form of generalization, identify training data that realizes it. It builds on singular learning theory and susceptibilities to connect data perturbations with changes in internal structure, and derives a minimum-norm data intervention via a patterning framework. The authors demonstrate the approach in two settings: (1) a small language model where reweighting data along a principal susceptibility direction accelerates or delays the emergence of an induction circuit, and (2) a parenthesis-balancing task where data perturbations shift the posterior toward one of two competing algorithms by manipulating local learning coefficients. These results establish a principled method to write internal structure by data, with potential implications for AI alignment and robust generalization, while noting current limitations in scale, computational cost, and online control extensions.

Abstract

Mechanistic interpretability aims to understand how neural networks generalize beyond their training data by reverse-engineering their internal structures. We introduce patterning as the dual problem: given a desired form of generalization, determine what training data produces it. Our approach is based on susceptibilities, which measure how posterior expectation values of observables respond to infinitesimal shifts in the data distribution. Inverting this linear response relationship yields the data intervention that steers the model toward a target internal configuration. We demonstrate patterning in a small language model, showing that re-weighting training data along principal susceptibility directions can accelerate or delay the formation of structure, such as the induction circuit. In a synthetic parentheses balancing task where multiple algorithms achieve perfect training accuracy, we show that patterning can select which algorithm the model learns by targeting the local learning coefficient of each solution. These results establish that the same mathematical framework used to read internal structure can be inverted to write it.

Figures (20)

• Figure 1: PC2 and induction patterns. Text from the training corpus highlighted in red and green based on PC2 value of the susceptibility vector of the 16 attention heads in the original model. Green indicates more positive values, while red indicates more negative. Note that the strongest red subsequences are rare biological terms, only highlighted red from their second appearance.
• Figure 2: Per-pattern susceptibilities for the induction pattern for each attention head are shown for the induction experiment, collected on the unmodified original training distribution. Note the $y$-axis range.
• Figure 3: We measure prefix matching scores (left) and previous token scores (right) from olsson2022context on the induction heads and previous token heads of each of the four seeds of models for each token weighting. The values over training for all 16 models are aggregated in these plots.
• Figure 4: Sequences of parentheses correspond to lattice paths: ( steps up, ) steps down. Sequences of parentheses that have an equal number of left and right parentheses map exactly onto those paths of diagonal steps which end on the $x$-axis. Sequences of parentheses that are correctly nested (i.e. are classified as True in our dataset) map exactly onto those paths of diagonal steps which both end on the $x$-axis and which remain at or above the $x$-axis at all other steps (left). Constructed examples of "almost nested" (center) and "almost equal" (right) samples are also given, see \ref{['appendix:bracket-datasets']} for precise definitions.
• Figure 5: Synthetically generated "almost nested" (left) and "almost equal" (right) samples visualized as heatmaps showing the number of Dyck paths crossing each lattice point. "Almost nested" paths climb before returning to near zero (but not zero); "almost equal" paths oscillate around zero height and end near but not at zero. Recall that we identify low OOD accuracy solutions with Equal-Count and high OOD accuracy solutions with Nested.

Theorems & Definitions (1)

• Definition 2.1