LLMs for XAI: Future Directions for Explaining Explanations

TL;DR

Explainability in ML remains challenging due to opaque models and difficult-to-interpret explanations. The paper proposes using Large Language Models to transform existing XAI explanations (e.g., SHAP-based outputs) into natural-language narratives, rather than directly explaining models. It introduces research directions (evaluation metrics, prompt design, model comparisons, fine-tuning, and external-data integration via retrieval-augmented generation) and reports first steps: zero-shot experiments comparing GPT-3.5 and GPT-4, plus a pilot user study. Findings show GPT-4 achieves higher soundness, completeness, and context-awareness, while narrative explanations are generally preferred over plot-based ones for understandability and informativeness, supporting the potential of LLM-based narratives to boost interpretability and trust. The work lays a roadmap for broader adoption and further investigation into context-aware, user-centered explanations in XAI.

Abstract

In response to the demand for Explainable Artificial Intelligence (XAI), we investigate the use of Large Language Models (LLMs) to transform ML explanations into natural, human-readable narratives. Rather than directly explaining ML models using LLMs, we focus on refining explanations computed using existing XAI algorithms. We outline several research directions, including defining evaluation metrics, prompt design, comparing LLM models, exploring further training methods, and integrating external data. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.