A Multimodal Automated Interpretability Agent

TL;DR

The paper tackles the challenge of scalable neural network interpretability by introducing MAIA, a multimodal agent that designs and runs iterative, tool-driven experiments to explain vision models. By composing a vision-language backbone with a Python API of interpretability tools, MAIA can describe neuron-level features, test causal hypotheses, and perform model auditing tasks such as removing spurious features and identifying biases. Empirical results show MAIA can produce predictive neuron descriptions that rival human experts and outperform certain baselines, with synthetic neurons providing ground-truth validation. While promising, the work also highlights limitations in tool reliability and the necessity of human oversight, pointing to a pathway for improving interpretability through better tools and more capable backbones.

Abstract

This paper describes MAIA, a Multimodal Automated Interpretability Agent. MAIA is a system that uses neural models to automate neural model understanding tasks like feature interpretation and failure mode discovery. It equips a pre-trained vision-language model with a set of tools that support iterative experimentation on subcomponents of other models to explain their behavior. These include tools commonly used by human interpretability researchers: for synthesizing and editing inputs, computing maximally activating exemplars from real-world datasets, and summarizing and describing experimental results. Interpretability experiments proposed by MAIA compose these tools to describe and explain system behavior. We evaluate applications of MAIA to computer vision models. We first characterize MAIA's ability to describe (neuron-level) features in learned representations of images. Across several trained models and a novel dataset of synthetic vision neurons with paired ground-truth descriptions, MAIA produces descriptions comparable to those generated by expert human experimenters. We then show that MAIA can aid in two additional interpretability tasks: reducing sensitivity to spurious features, and automatically identifying inputs likely to be mis-classified.

Figures (13)

• Figure 1: maia framework.maia autonomously conducts experiments on other systems to explain their behavior.
• Figure 2: maia experiments for labeling neurons.maia iteratively writes programs that compose common interpretability tools to conduct experiments on other systems. At each step, maia autonomously makes and updates hypotheses in light of experimental outcomes, showing sophisticated scientific reasoning capabilities. Generated code is executed with a Python interpreter and the outputs (shown above, neuron activation values overlaid in white, masks thresholded at 0.95 percentile of activation maps) are returned to maia.
• Figure 3: Predictive evaluation protocol. We compare neuron labeling methods by assessing how well their labels predict neuron activation values on unseen data. For each neuron we perform the following steps: (a) An LM uses candidate neuron labels to generate a set of image prompts that should maximally/neutrally activate the neuron. (b) All prompts (positive and neutral) from all methods are combined into one dataset. (c) For each labeling method, a new LM selects prompts from the Prompt Dataset that are likely to produce maximal and neutral neuron activations, if that label were accurate. (d) A text-to-image model generates all corresponding images, and the average activation values for positive and neutral images are recorded. A predictive neuron label will produce exemplars with maximally positive activations relative to the neutral baseline.
• Figure 4: Predictive evaluation results. The average positive activation values ("+") for maia labels outperform milan and are comparable to human descriptions for both real and synthetic neurons. Neutral activations ("-") are comparable across methods.
• Figure 5: Synthetic neuron implementation. Segmentation of input images is performed by an open-set concept detector with text guidance specifying ground-truth neuron selectivity. Synthetic neurons return masked images and synthetic activation values corresponding to the probability a concept is present in the image.