Properties and Challenges of LLM-Generated Explanations
Jenny Kunz, Marco Kuhlmann
TL;DR
The paper investigates the properties of explanations generated by large language models (LLMs) and how they align with characteristics of human explanations. Using Alpaca data derived from GPT-4 annotations, it conducts a human analysis across 200 instructions to assess whether explanations exist and which properties they exhibit. The findings show that LLM explanations frequently display selectivity and illustrative elements, while subjectivity and misleading explanations are relatively rare, suggesting alignment effects from pre-training and instruction fine-tuning. The authors discuss the implications for safety, trustworthiness, troubleshooting, and knowledge discovery, and emphasize that the suitability of explanations depends on user goals and contexts, while noting limitations related to dataset choice, language, and model specificity.
Abstract
The self-rationalising capabilities of large language models (LLMs) have been explored in restricted settings, using task/specific data sets. However, current LLMs do not (only) rely on specifically annotated data; nonetheless, they frequently explain their outputs. The properties of the generated explanations are influenced by the pre-training corpus and by the target data used for instruction fine-tuning. As the pre-training corpus includes a large amount of human-written explanations "in the wild", we hypothesise that LLMs adopt common properties of human explanations. By analysing the outputs for a multi-domain instruction fine-tuning data set, we find that generated explanations show selectivity and contain illustrative elements, but less frequently are subjective or misleading. We discuss reasons and consequences of the properties' presence or absence. In particular, we outline positive and negative implications depending on the goals and user groups of the self-rationalising system.