ALMANACS: A Simulatability Benchmark for Language Model Explainability
Edmund Mills, Shiye Su, Stuart Russell, Scott Emmons
TL;DR
ALMANACS introduces a fully automated simulatability benchmark to evaluate language-model explainability, framing explanations via an explainer $\\mathcal{E}$ and evaluating them with a predictor $\\mathcal{P}$ on distributionally shifted Yes/No scenarios across 12 safety topics. It tests Counterfactuals, Rationalizations, Attention, and Integrated Gradients using GPT-4 as the automated predictor, reporting evaluation with $KLDiv$, $TVDist$, and Spearman metrics. Across two LLMs (e.g., $flan-alpaca-gpt4-xl$ and $vicuna-7b-v1.3$), results show that none of the explanation methods consistently improves simulatability on average, underscoring the challenge of creating explanations that aid predictive reasoning. The study also validates the GPT-4 predictor’s capabilities and finds broad alignment with human predictors, though some subtasks show divergence, highlighting ALMANACS as a complementary tool rather than a replacement for human evaluation. Overall, ALMANACS offers a scalable, automated framework for rapid screening of explainability methods and contributes to understanding the limits of current explanations in aiding model simulatability $\left(\text{explanations}\not\Rightarrow\ \text{predictive gains on average}\right)$.
Abstract
How do we measure the efficacy of language model explainability methods? While many explainability methods have been developed, they are typically evaluated on bespoke tasks, preventing an apples-to-apples comparison. To help fill this gap, we present ALMANACS, a language model explainability benchmark. ALMANACS scores explainability methods on simulatability, i.e., how well the explanations improve behavior prediction on new inputs. The ALMANACS scenarios span twelve safety-relevant topics such as ethical reasoning and advanced AI behaviors; they have idiosyncratic premises to invoke model-specific behavior; and they have a train-test distributional shift to encourage faithful explanations. By using another language model to predict behavior based on the explanations, ALMANACS is a fully automated benchmark. While not a replacement for human evaluations, we aim for ALMANACS to be a complementary, automated tool that allows for fast, scalable evaluation. Using ALMANACS, we evaluate counterfactual, rationalization, attention, and Integrated Gradients explanations. Our results are sobering: when averaged across all topics, no explanation method outperforms the explanation-free control. We conclude that despite modest successes in prior work, developing an explanation method that aids simulatability in ALMANACS remains an open challenge.