BlackboxNLP-2025 MIB Shared Task: Improving Circuit Faithfulness via Better Edge Selection

TL;DR

This work tackles circuit discovery in mechanistic interpretability under the Mechanistic Interpretability Benchmark (MIB) by introducing three edge-selection enhancements: bootstrapped confidence filtering to stabilize edge signs, a Positive-Negative Ratio (PNR) to balance edge contributions, and an Integer Linear Programming (ILP) formulation for globally optimal circuit construction. Built on Edge Attribution Patching with Integrated Gradients (EAP-IG) scores, these methods yield more faithful circuits and outperform greedy baselines across multiple models and tasks, with comprehensive ablations validating the contributions. The approach demonstrates that principled edge selection can improve faithfulness but faces scalability and task-specific tuning challenges, pointing to avenues for improved attribution methods and scalable optimization. Overall, the paper advances circuit discovery by combining robust statistical filtering, controlled edge-sign balancing, and global optimization to enhance interpretability in large models.

Abstract

One of the main challenges in mechanistic interpretability is circuit discovery, determining which parts of a model perform a given task. We build on the Mechanistic Interpretability Benchmark (MIB) and propose three key improvements to circuit discovery. First, we use bootstrapping to identify edges with consistent attribution scores. Second, we introduce a simple ratio-based selection strategy to prioritize strong positive-scoring edges, balancing performance and faithfulness. Third, we replace the standard greedy selection with an integer linear programming formulation. Our methods yield more faithful circuits and outperform prior approaches across multiple MIB tasks and models. Our code is available at: https://github.com/technion-cs-nlp/MIB-Shared-Task.