Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations

TL;DR

The paper investigates whether verbalized self-explanations from large language models enhance users' ability to predict model behavior in counterfactual scenarios. It extends counterfactual simulatability by comparing pragmatics-based perturbations rooted in Gricean maxims with LLM-generated counterfactuals, using StrategyQA as the testbed. By evaluating both human participants and LLM judges, the study shows that explanations improve simulation accuracy, though the magnitude and stability of gains depend on perturbation type and judge strength, with explanations also reducing anchoring to the original answer. Qualitative human-rationales analyses reveal that explanations are most helpful when they articulate explicit decision criteria that align with the model’s reasoning, while ceiling effects and misleading but persuasive explanations can limit benefits. The work highlights the importance of perturbation design in explanation evaluations and provides open resources to support transparent, reproducible research in explainable AI.

Abstract

Large Language Models (LLMs) can produce verbalized self-explanations, yet prior studies suggest that such rationales may not reliably reflect the model's true decision process. We ask whether these explanations nevertheless help users predict model behavior, operationalized as counterfactual simulatability. Using StrategyQA, we evaluate how well humans and LLM judges can predict a model's answers to counterfactual follow-up questions, with and without access to the model's chain-of-thought or post-hoc explanations. We compare LLM-generated counterfactuals with pragmatics-based perturbations as alternative ways to construct test cases for assessing the potential usefulness of explanations. Our results show that self-explanations consistently improve simulation accuracy for both LLM judges and humans, but the degree and stability of gains depend strongly on the perturbation strategy and judge strength. We also conduct a qualitative analysis of free-text justifications written by human users when predicting the model's behavior, which provides evidence that access to explanations helps humans form more accurate predictions on the perturbed questions.

Figures (5)

• Figure 1: Pipeline for evaluating the usefulness of LLM self-explanations based on different counterfactual question perturbations. For each original $Q$ and model prediction, we measure users’ ability to predict model behavior on different counterfactual question sets $Q'$, with and without access to the model’s self-explanations.
• Figure 2: Counterfactual simulation pipeline for evaluating how human and LLM judges predict model behavior from explanations and counterfactual question sets.
• Figure 3: Radar charts summarizing LLM-as-a-judge acc_improvement on different counterfactual question sets. The upper and lower panels show results for a weaker and a stronger evaluator, respectively, each using combinations of chain-of-thought (cot) and post-hoc explanations for three underlying models to be explained.
• Figure 4: User study interface: instruction for phase 1.
• Figure 5: User study interface: instruction for phase 2.