Explaining black box text modules in natural language with language models

TL;DR

SASC proposes a black-box, two-step framework to generate natural-language explanations for text modules by grounding explanations in phrases that elicit strong responses and validating them with synthetic data. It demonstrates reliable recovery of ground-truth explanations on synthetic modules, yields explanations for BERT transformer factors comparable to human annotations, and extends to explaining fMRI voxel responses to language. The method leverages a pre-trained helper LLM for summarization and generation, while its synthetic scoring provides a reliability measure for explanations, enabling principled interpretability across NLP models and neuroscience data. This approach promises mechanistic insight and auditability for large language systems and language-responsive brain areas, with broad potential for scientific and applied use.

Abstract

Large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks. However, their rapid proliferation and increasing opaqueness have created a growing need for interpretability. Here, we ask whether we can automatically obtain natural language explanations for black box text modules. A "text module" is any function that maps text to a scalar continuous value, such as a submodule within an LLM or a fitted model of a brain region. "Black box" indicates that we only have access to the module's inputs/outputs. We introduce Summarize and Score (SASC), a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is. We study SASC in 3 contexts. First, we evaluate SASC on synthetic modules and find that it often recovers ground truth explanations. Second, we use SASC to explain modules found within a pre-trained BERT model, enabling inspection of the model's internals. Finally, we show that SASC can generate explanations for the response of individual fMRI voxels to language stimuli, with potential applications to fine-grained brain mapping. All code for using SASC and reproducing results is made available on Github.

Figures (10)

• Figure 1: SASC pipeline for obtaining a natural language explanation given a module f. (i) SASC first generates candidate explanations (using a pre-trained LLM) based on the ngrams that elicit the most positive response from $f$. (ii) SASC then evaluates each candidate explanation by generating synthetic data based on the explanation and testing the response of $f$ to the data.
• Figure 2: Cumulative accuracy at recovering the ground truth explanation increases as a function of explanation score. Error bars show standard error of the mean.
• Figure 3: Explanation score for BERT (blue) and fMRI (orange) modules. As the BERT layer increases, the explanation score tends to decrease, implying modules are harder to explain with SASC. Across regions, explanation scores for fMRI voxel modules are generally lower than scores for BERT modules in early layers and comparable to scores for the final layers. Boxes show the median and interquartile range. ROI abbreviations: premotor ventral hand area (PMvh), anterior temporal face patch (ATFP), auditory cortex (AC), parietal operculum (PO), inferior frontal sulcus face patch (IFSFP), Broca's area (Broca).
• Figure A1: The BERT score between generated explanation and groundtruth explanation generally increases as the size of the helper LLM for summarization/generation increases. Models are accessed via the OpenAI API (text-ada-001, text-babbage-001, text-curie-001, text-davinci-001, all accessed on Feb. 2023) and are in order of increasing size. BERT score for each module is computed as the maximum over the 5 generated explanations.
• Figure A2: Explanation BERT score for the 54 synthetic datasets as a function of corpus size. Performance plateaus around 100,000 ngrams. Corpus is created by randomly subsampling the unique trigrams in the WikiText dataset merity2016pointer. Gray dotted line shows the result when evaluating on dataset-specific corpuses, as in the Default setting in \ref{['tab:recovery_results']}.