Enhancing XAI Narratives through Multi-Narrative Refinement and Knowledge Distillation

TL;DR

This work presents Multi-Narrative Refinement (MNR), a two-stage, knowledge-distillation framework that enables small language models to generate high-quality counterfactual narratives for tabular data. By producing multiple drafts (Draft Narrative Generator) and a subsequent Refiner to synthesize and correct them, the approach substantially improves feature-faithfulness metrics (AvgFF, PFF, TF) and narrative quality while reducing compute (memory, time, energy) compared to using a large teacher model alone. The authors provide a formal background for counterfactual explanations and narratives, introduce a structured evaluation protocol combining automated metrics with human judgments, and demonstrate substantial gains on Adult and Titanic datasets. The work offers practical pathways for deploying explanation-generation in real-world, policy-driven contexts, balancing interpretability, efficiency, and robustness while acknowledging social risks and the need for careful governance.

Abstract

Explainable Artificial Intelligence has become a crucial area of research, aiming to demystify the decision-making processes of deep learning models. Among various explainability techniques, counterfactual explanations have been proven particularly promising, as they offer insights into model behavior by highlighting minimal changes that would alter a prediction. Despite their potential, these explanations are often complex and technical, making them difficult for non-experts to interpret. To address this challenge, we propose a novel pipeline that leverages Language Models, large and small, to compose narratives for counterfactual explanations. We employ knowledge distillation techniques along with a refining mechanism to enable Small Language Models to perform comparably to their larger counterparts while maintaining robust reasoning abilities. In addition, we introduce a simple but effective evaluation method to assess natural language narratives, designed to verify whether the models' responses are in line with the factual, counterfactual ground truth. As a result, our proposed pipeline enhances both the reasoning capabilities and practical performance of student models, making them more suitable for real-world use cases.

Figures (4)

• Figure 1: Overview of the dataset generation process for the draft narrative generation step and the refiner step using knowledge distillation. Given a factual instance $\boldsymbol{x}$ with its features (e.g., age, work class, education, etc.), a counterfactual generator $g(\boldsymbol{x})$ modifies the instance to create a counterfactual example $\boldsymbol{x}'$, altering specific attributes (e.g., age, work class) to yield a different predicted outcome (e.g., income = 0) (red zone). The factual and counterfactual instances are then integrated into the prompt that is fed into the teacher model, which produces the narrative for the draft narrative generation step (yellow zone) and for the refiner step (purple zone). For the draft narrative generation step, the response is collected and put along with the prompt $p_{draft}$ in the new dataset $D_{draft}$. Concerning the dataset for the refiner step, we generate $N=3$ draft responses and integrate all three into the prompt $p_{refiner}$. Finally, we feed the prompt into the teacher, collect the response, and put the newly formed pair $p_{refiner}, response$ into the dataset $D_{refiner}$.
• Figure 2: Multi-Narrative Refinement pipeline schema
• Figure 3: GPU power comparison of three techniques for narrative generation. Our approach (MNR pipeline) uses Qwen2.5-0.5B-I. as draft generator $\mathcal{M}$ and Qwen2.5-3B-I. as refiner $\mathcal{R}$. DeepSeek-R1-D.-Qwen-32B is the teacher model, while Qwen2.5-0.5B-I is the fine-tuned model without refiner. Solid lines show average power; shaded areas denote standard deviation.
• Figure 4: Right side: prompt used to refine multile narratives. Left side: prompt to generate the draft explanations.

Theorems & Definitions (2)

• Definition 1: The Counterfactual Explanation Problem
• Definition 2: The Counterfactual Narrative Generation Problem