A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

TL;DR

This work tackles the challenge of trusting LLM self-explanations by introducing Normalized Simulatability Gain (NSG), a predictor-based metric that measures the predictive information embedded in explanations about an LLM's decision criteria. By grounding counterfactuals in real data and evaluating 18 frontier models across 7 datasets with 7,000 question-counterfactual pairs, the authors demonstrate that self-explanations significantly improve the ability to predict model behavior (NSG ranging roughly from 11% to 36.5%), and often outperform explanations produced by external models. The study also reveals that a non-negligible 5–15% of self-explanations are egregiously unfaithful, and that faithfulness can vary across datasets and model families, with limited returns from simply increasing reasoning strength. Importantly, self-explanations show a privileged self-knowledge advantage over cross-model explanations, suggesting that internal access to reasoning traces provides unique predictive value. Overall, NSG offers a scalable, non-adversarial framework for auditing explanation faithfulness and informing AI safety practices in frontier LLMs.

Abstract

LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce Normalized Simulatability Gain (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are egregiously misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.

Figures (31)

• Figure 1: Faithful explanations should reveal decision-making criteria. An LLM assesses two patients for heart disease. The patients' profiles differ only in age. The LLM switches answers, indicating age is a determining factor. A faithful explanation should mention the influence of age.
• Figure 2: Self-explanations encode valuable information about models' decision-making criteria. We introduce Normalized Simulatability Gain, a metric that measures the predictive information self-explanations provide (Section \ref{['sec:counterfactual_sim']}). Across 18 leading open-weight and proprietary models, including the Qwen 3, Gemma 3, GPT-5, Claude 4.5, and Gemini 3 families, we find self-explanations often faithfully explain models' decision-making criteria (with significant room for further improvement). Bars show predictor accuracy without access to explanations (dark) and with access to explanations (hashed). Accuracy is averaged across five predictor models: gpt-oss-20b, Qwen-3-32B, gemma-3-27b-it, GPT-5 mini, gemini-3-flash. For predictor-specific results, see Appendix \ref{['app:sec:predictor_model_stability']}. Error bars show 95% bootstrap CIs.
• Figure 3: Operationalizing faithfulness with NSG. The model under evaluation (the reference model) produces both an answer and accompanying explanation for an input question (illustrated here with the Heart Disease dataset). A separate predictor model uses the explanation to simulate how the reference model would respond to a related counterfactual. The metric is based on the principle that more faithful explanations enable more accurate counterfactual simulation. In the top branch the explanation helps predictive performance, in the bottom branch the explanation does not help.
• Figure 4: Representative questions in the dataset. Left: (upper) Employee Attrition, (lower) Heart Disease classification. Right: (upper) Breast Cancer Recurrence, (lower) Income Prediction. The full dataset contains questions on diabetes classification, trolley problems, and bank marketing outcomes.
• Figure 5: Mixed trends between model scale and faithfulness. The Qwen 3 family shows a clear monotonic relationship with model scale and there is an upward trend for Gemma 3, but this does not hold for proprietary models. Error bars show 95% CIs.