An Agentic Approach to Generating XAI-Narratives

Abstract

Explainable AI (XAI) research has experienced substantial growth in recent years. Existing XAI methods, however, have been criticized for being technical and expert-oriented, motivating the development of more interpretable and accessible explanations. In response, large language model (LLM)-generated XAI narratives have been proposed as a promising approach for translating post-hoc explanations into more accessible, natural-language explanations. In this work, we propose a multi-agent framework for XAI narrative generation and refinement. The framework comprises the Narrator, which generates and revises narratives based on feedback from multiple Critic Agents on faithfulness and coherence metrics, thereby enabling narrative improvement through iteration. We design five agentic systems (Basic Design, Critic Design, Critic-Rule Design, Coherent Design, and Coherent-Rule Design) and systematically evaluate their effectiveness across five LLMs on five tabular datasets. Results validate that the Basic Design, the Critic Design, and the Critic-Rule Design are effective in improving the faithfulness of narratives across all LLMs. Claude-4.5-Sonnet on Basic Design performs best, reducing the number of unfaithful narratives by 90% after three rounds of iteration. To address recurrent issues, we further introduce an ensemble strategy based on majority voting. This approach consistently enhances performance for four LLMs, except for DeepSeek-V3.2-Exp. These findings highlight the potential of agentic systems to produce faithful and coherent XAI narratives.

Figures (14)

• Figure 1: The baseline narratives are fed into the Faithful Evaluator and the Coherence Agent in the beginning. Based on the Faithful Evaluator's output, the Faithful Critic offers revision suggestions to the Narrator. On the other side, the Coherence Agent identifies coherence-related issues and provides revision suggestions to the Narrator. The Narrator then generates an updated version of the narrative and continues the iteration until the stopping criterion is met.
• Figure 2: Left: This is an example of the base prompt for the Narrator. The instance is from the Student dataset. Right: The Narrator generates a narrative based on the given SHAP input. However, in this example, it injects multiple faithfulness-related issues, including the wrong feature value on "goout" (being 5 times instead of 4 times in the given SHAP input), the wrong feature sign on "Walc" (being "positive" instead of "negative"), and the interchanged feature importance ranking between "goout" and "failures". These issues are marked in red.
• Figure 3: Left: The Faithful Evaluator's prompt that we use to extract information from a narrative. The narrative from Figure \ref{['NA']} is fed into the prompt. Middle: The Faithful Evaluator extracts the information into a dictionary, including the rank, sign, and value information for each feature. This is an intermediate output, which can be reused by the Faithful Critic. Right: The output from the Faithful Evaluator. The errors contained in the narrative are captured and reported by the Faithful Evaluator, shown in red.
• Figure 4: Upper Left: The instance's SHAP input table. Below Left: The extraction dictionary generated by the Faithful Evaluator. All mistakes are successfully extracted and shown in red. Right: The output from the Faithful Critic. It refers to the original SHAP table and outputs directional feedback, guiding the Narrator on how to revise.
• Figure 5: Two examples of the feedback from the Coherence Agent. Left: Two narratives that contain coherence-related issues. Problems are highlighted in red. Right: The corresponding feedback from the Coherence Agent.