Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models

TL;DR

This work frames mechanistic interpretability (MI) within a standardized, reproducible benchmark (MIB) and documents the BlackboxNLP 2025 Shared Task, which comprises two tracks: circuit localization and causal variable localization. Circuit localization evaluates how well methods identify minimal, causally influential components using metrics like CPR and CMD derived from a faithfulness measure, while causal variable localization uses interchange interventions to assess whether featurized variables faithfully mirror causal roles. Across three teams and ten methods, the task demonstrates that ensembling and regularization improve circuit discovery and that non-linear featurizers can significantly enhance variable localization, albeit with caveats about whether the learned features correspond to genuine model mechanisms. The findings underscore practical strategies for MI research (ensemble approaches, robust edge selection, and nonlinear featurization) and establish a platform for ongoing, comparable progress toward interpretable language models, with the MIB leaderboard remaining open for future submissions. Specifically, key formulations such as $f(\mathcal{C}, \mathcal{N}; m) = \frac{m(\mathcal{C}) - m(\varnothing)}{m(\mathcal{N}) - m(\varnothing)}$, CPR as the area under the faithfulness curve, and CMD as the deviation from perfect faithfulness summarize the quantitative backbone of the evaluation.

Abstract

Mechanistic interpretability (MI) seeks to uncover how language models (LMs) implement specific behaviors, yet measuring progress in MI remains challenging. The recently released Mechanistic Interpretability Benchmark (MIB; Mueller et al., 2025) provides a standardized framework for evaluating circuit and causal variable localization. Building on this foundation, the BlackboxNLP 2025 Shared Task extends MIB into a community-wide reproducible comparison of MI techniques. The shared task features two tracks: circuit localization, which assesses methods that identify causally influential components and interactions driving model behavior, and causal variable localization, which evaluates approaches that map activations into interpretable features. With three teams spanning eight different methods, participants achieved notable gains in circuit localization using ensemble and regularization strategies for circuit discovery. With one team spanning two methods, participants achieved significant gains in causal variable localization using low-dimensional and non-linear projections to featurize activation vectors. The MIB leaderboard remains open; we encourage continued work in this standard evaluation framework to measure progress in MI research going forward.

Figures (1)

• Figure 1: Overview of the evaluation method for each track in MIB. The circuit localization track requires uploading multiple circuits or importance scores for each component; we evaluate by taking the area under the faithfulness curve across circuit sizes. The causal variable localization track requires uploading a featurizer and location; we evaluate by intervening on the concept in the featurized space and measuring whether the model's behavior changes in the expected way. Figure reproduced from muellermib with permission.