Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to use and interpret activation patching

Stefan Heimersheim, Neel Nanda

TL;DR

Activation patching is presented as a practical mechanistic interpretability tool, with careful attention to how to design experiments, interpret results, and choose metrics. The authors advocate using denoising and noising patching over ablations, and outline exploratory versus confirmatory regimes, granularity choices, and path patching for testing direct versus mediated interactions. Through concrete circuit examples (AND vs OR) and a Nobel Prize-inspired prompt, the work illustrates sufficiency versus necessity and highlights the impact of corrupted prompts on inferred circuits. The paper emphasizes metric selection and warns against pitfalls, recommending multiple continuous metrics (e.g., logit difference, logprob) and considering a Pareto frontier of circuit size versus recovered performance to guide interpretation and reporting.

Abstract

Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.

How to use and interpret activation patching

TL;DR

Activation patching is presented as a practical mechanistic interpretability tool, with careful attention to how to design experiments, interpret results, and choose metrics. The authors advocate using denoising and noising patching over ablations, and outline exploratory versus confirmatory regimes, granularity choices, and path patching for testing direct versus mediated interactions. Through concrete circuit examples (AND vs OR) and a Nobel Prize-inspired prompt, the work illustrates sufficiency versus necessity and highlights the impact of corrupted prompts on inferred circuits. The paper emphasizes metric selection and warns against pitfalls, recommending multiple continuous metrics (e.g., logit difference, logprob) and considering a Pareto frontier of circuit size versus recovered performance to guide interpretation and reporting.

Abstract

Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.
Paper Structure (26 sections, 3 figures, 1 table)

This paper contains 26 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. What is activation patching?
  3. How is this related to ablation?
  4. An example
  5. What is this document about
  6. What kind of patching experiments should you run?
  7. Exploratory and confirmatory experiments
  8. Which components should you patch
  9. Noising and Denoising
  10. Example: AND gate vs OR gate
  11. AND circuit:
  12. OR circuit:
  13. Comparison to ablations and other patching techniques
  14. Choosing corrupted prompts
  15. How do you interpret patching evidence?
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Toy "Nobel Peace Price" circuit
  • Figure 2: Illustration of different metrics for an example patching experiment with GPT-2 medium.
  • Figure :