Investigating Training and Generalization in Faithful Self-Explanations of Large Language Models

TL;DR

The paper tackles the persistent problem of faithfulness in LLM self-explanations by constructing pseudo-faithful explanations across attribution, redaction, and counterfactual styles and training instruction-tuned models with these signals. Using three tasks (Sentiment140, SNLI, AGNews) and two model scales (Tulu-2 7B, 13B), it demonstrates that training with pseudo-faithful explanations improves the proportion of faithful self-explanations across styles and tasks. The study also shows partial generalization to unconstrained multi-word explanations and unseen tasks, and observes cross-style transfer of faithfulness improvements, suggesting a broader enhancement of self-explanation capability. Limitations include reliance on one-word constraints, dataset scope, and access to training data, but the results offer a practical path toward more trustworthy LLM explanations.

Abstract

Large language models have the potential to generate explanations for their own predictions in a variety of styles based on user instructions. Recent research has examined whether these self-explanations faithfully reflect the models' actual behavior and has found that they often lack faithfulness. However, the question of how to improve faithfulness remains underexplored. Moreover, because different explanation styles have superficially distinct characteristics, it is unclear whether improvements observed in one style also arise when using other styles. This study analyzes the effects of training for faithful self-explanations and the extent to which these effects generalize, using three classification tasks and three explanation styles. We construct one-word constrained explanations that are likely to be faithful using a feature attribution method, and use these pseudo-faithful self-explanations for continual learning on instruction-tuned models. Our experiments demonstrate that training can improve self-explanation faithfulness across all classification tasks and explanation styles, and that these improvements also show signs of generalization to the multi-word settings and to unseen tasks. Furthermore, we find consistent cross-style generalization among three styles, suggesting that training may contribute to a broader improvement in faithful self-explanation ability.

Figures (4)

• Figure 1: Examples of one-word-constrained self-explanations and faithfulness evaluation for each explanation style. Self-explanations are generated in the same session as the classification task: Attribution and Redaction require listing and redacting the most important input words affecting the prediction, respectively, while Counterfactual requires editing the input text so that the predicted label will flip. Faithfulness evaluation involves a separate session, in which a self-explanation is considered faithful if editing the input according to it indeed flips the prediction.
• Figure 2: Evaluation of the generalization to the multi-word setting (Section \ref{['subsec: Generalization to Multi-Word Explanation Styles']}) on the Sentiment140 dataset. We report the proportion of faithful self-explanations for each number of words that are used in the self-explanations for each style. Data plots marked with "$\times$" indicate that the number of evaluation instances is less than 50.
• Figure 3: Evaluation of the generalization across different classification tasks. For each training-evaluation task pair, we measure the faithfulness score gain before and after training with self-explanations, defined as the increase or decrease in the proportion of faithful self-explanations. Results are reported using the Tulu-2 13B model.
• Figure 4: Evaluation of generalization across explanation styles. Each value represents the faithfulness score obtained under a given evaluation style, and each training condition specifies the explanation style used for training. Values marked with "*" indicate that the number of evaluation instances is less than 50. Results are reported using the Tulu-2 13B model.