Explaining the Explainer: Understanding the Inner Workings of Transformer-based Symbolic Regression Models

TL;DR

This work investigates mechanistic interpretability for transformer-based symbolic regression by introducing PATCHES, an evolutionary circuit-discovery algorithm that identifies compact, causal circuits responsible for producing symbolic operators. The authors extract 28 circuits, of which 13 are faithful, complete, and minimal, with mean patching delivering the most reliable causal isolation across experiments. They demonstrate that direct logit attribution and probing classifiers often capture correlational rather than causal features, emphasizing the value of function-level evaluation for circuit discovery. Overall, the study establishes SR as a promising domain for mechanistic interpretability and provides a principled, model-agnostic pipeline for circuit discovery that can generalize beyond the current setting.

Abstract

Following their success across many domains, transformers have also proven effective for symbolic regression (SR); however, the internal mechanisms underlying their generation of mathematical operators remain largely unexplored. Although mechanistic interpretability has successfully identified circuits in language and vision models, it has not yet been applied to SR. In this article, we introduce PATCHES, an evolutionary circuit discovery algorithm that identifies compact and correct circuits for SR. Using PATCHES, we isolate 28 circuits, providing the first circuit-level characterisation of an SR transformer. We validate these findings through a robust causal evaluation framework based on key notions such as faithfulness, completeness, and minimality. Our analysis shows that mean patching with performance-based evaluation most reliably isolates functionally correct circuits. In contrast, we demonstrate that direct logit attribution and probing classifiers primarily capture correlational features rather than causal ones, limiting their utility for circuit discovery. Overall, these results establish SR as a high-potential application domain for mechanistic interpretability and propose a principled methodology for circuit discovery.

Figures (18)

• Figure 1: The PATCHES Framework.Left: Model schematic: equation samples and previously decoded samples are processed by layers (rows) and components (columns) to predict the target token (e.g., $\sin$). Center: Discovery loop. We (1) configure the patching strategy, (2-3) sample candidate masks via CMA-ES, and (4) refine the search distribution based on a fitness trade-off between performance and circuit size. Right: Validation criteria based on the best circuit (green). $\mathcal{M}$: Full model; $\mathcal{M}_C$: Only circuit active; $\mathcal{M}_{\downarrow C}$: Only circuit complement active; $\mathcal{M}_{C_{\downarrow c_i}}$: Only circuit active without circuit component $c_i$.
• Figure 2: Diagram illustrating three patching strategies applied to the formula $x_1+\sin(x_2)$, target being $\sin$. The patching strategy is selected manually. Corresponding patches are cached, averaged, and used to modify the selected parts of the model; in this case, Layer 1's feedforward block and Layer 2's Head 1.
• Figure 3: Confusion plots illustrating the diversity of circuits identified in the model.(a) Overlap percentages between operators; the diagonal (black) indicates circuit length. RF: Resample Functional; RM: Resample Model; MF Mean Functional; MM: Mean Model; RFSTR: Resample Functional STR (b) Usage frequency of components in the selected circuits. Lx.x denotes layer x.x; OUT the output projection; MLP the multilayer perceptron block; Hx attention head x. Full tables and figures in \ref{['Additional Verification Experiments']}.
• Figure 4: Probing and Iterative Patching Results. RF: Resample Functional, RM: Resample Model, MF Mean Functional, MM: Mean Model, RFSTR: Resample Functional STR. (a) Probe accuracy comparisons between circuit and complement components. Full Table in \ref{['tab:probing']}. (b) Circuit length comparison between patches and Iterative Patching. / indicate incorrect and correct circuits respectively. Full Table in \ref{['Iterative Patching: Additional Results']}.
• Figure 5: Direct attribution and faithfulness evaluation. (a) Change in logit score vs. sin mean when patching individual heads, averaged over 100 samples. Lx.x denotes layer x.x; OUT the output projection; MLP the multilayer perceptron block; Hx attention head x. (b) Faithfulness evaluation of sin mean importance ranking; thresholds (dashed lines) from \ref{['tab:baseline+circuit_results']}.