From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality

TL;DR

This study systematically examines factors shaping LLM-generated natural language explanations (NLEs) of time-series forecasts by employing a factorial Design of Experiments across four forecasting models (XGBoost, Random Forest, MLP, SARIMAX), three XAI conditions (SHAP, LIME, and none), three LLMs (GPT-4o, Llama-3, DeepSeek-R1), and eight prompting strategies, yielding 660 explanations evaluated with a dual-LLM G-Eval framework across four criteria. Key findings show that the LLM choice dominates explanation quality (DeepSeek-R1 performing best, at higher token cost), while including XAI outputs offers only modest gains (mostly for Expert Relevancy) and SHAP/LIME are largely indistinguishable. An interpretability paradox emerges: the inherently interpretable SARIMAX model produces lower NLE quality than ML models when explained, suggesting LLMs struggle to translate statistical coefficients into accessible narratives. Zero-shot prompting often matches more complex prompting strategies, with self-consistency offering marginal gains at substantially higher cost. Overall, the work challenges the assumption that more sophisticated XAI outputs automatically enhance explanations, highlighting the central role of the LLM's narrative translation and prompting strategy in NLE quality.

Abstract

Explainable AI (XAI) methods like SHAP and LIME produce numerical feature attributions that remain inaccessible to non expert users. Prior work has shown that Large Language Models (LLMs) can transform these outputs into natural language explanations (NLEs), but it remains unclear which factors contribute to high-quality explanations. We present a systematic factorial study investigating how Forecasting model choice, XAI method, LLM selection, and prompting strategy affect NLE quality. Our design spans four models (XGBoost (XGB), Random Forest (RF), Multilayer Perceptron (MLP), and SARIMAX - comparing black-box Machine-Learning (ML) against classical time-series approaches), three XAI conditions (SHAP, LIME, and a no-XAI baseline), three LLMs (GPT-4o, Llama-3-8B, DeepSeek-R1), and eight prompting strategies. Using G-Eval, an LLM-as-a-judge evaluation method, with dual LLM judges and four evaluation criteria, we evaluate 660 explanations for time-series forecasting. Our results suggest that: (1) XAI provides only small improvements over no-XAI baselines, and only for expert audiences; (2) LLM choice dominates all other factors, with DeepSeek-R1 outperforming GPT-4o and Llama-3; (3) we observe an interpretability paradox: in our setting, SARIMAX yielded lower NLE quality than ML models despite higher prediction accuracy; (4) zero-shot prompting is competitive with self-consistency at 7-times lower cost; and (5) chain-of-thought hurts rather than helps.

Figures (3)

• Figure 1: Overview of our Prediction-to-NLE pipeline: Models generate predictions, XAI methods (SHAP/LIME) compute feature attributions, and LLMs transform these numerical outputs into narratives accessible to lay users.
• Figure 2: Factorial experimental design with four factors and their levels (top), and five G-Eval evaluation criteria (bottom).
• Figure 3: Test-set predictions vs. actual weekly household energy consumption (kWh). XGBoost (R$^2$=0.69) captures temporal patterns effectively; Random Forest (R$^2$=0.37) shows moderate tracking; SARIMAX (R$^2$=0.55) handles seasonality but misses peaks; MLP (R$^2$=0.21) exhibits high variance with poor trend-following.