Explainable AI Components for Narrative Map Extraction

TL;DR

The paper presents an Explainable AI system for narrative map extraction that delivers explanations at multiple levels of abstraction: low-level topical clusters, connection explanations leveraging SHAP values, and high-level storyline names plus important event detection. It evaluates the system via an insight-based user study on 160 Cuban protest articles with 10 participants, finding that the explanations, especially connection labels and important events, enhance user trust and sensemaking. The approach bridges document embeddings, coherence-based narrative construction, and visualization to support reliable human-AI collaboration, while acknowledging limitations such as potential information overload and the need for baselines. The work contributes practical designs and empirical evidence for trustworthy narrative extraction, and makes the system publicly available (GitHub) to foster adoption and further research.

Abstract

As narrative extraction systems grow in complexity, establishing user trust through interpretable and explainable outputs becomes increasingly critical. This paper presents an evaluation of an Explainable Artificial Intelligence (XAI) system for narrative map extraction that provides meaningful explanations across multiple levels of abstraction. Our system integrates explanations based on topical clusters for low-level document relationships, connection explanations for event relationships, and high-level structure explanations for overall narrative patterns. In particular, we evaluate the XAI system through a user study involving 10 participants that examined narratives from the 2021 Cuban protests. The analysis of results demonstrates that participants using the explanations made the users trust in the system's decisions, with connection explanations and important event detection proving particularly effective at building user confidence. Survey responses indicate that the multi-level explanation approach helped users develop appropriate trust in the system's narrative extraction capabilities. This work advances the state-of-the-art in explainable narrative extraction while providing practical insights for developing reliable narrative extraction systems that support effective human-AI collaboration.

Figures (3)

• Figure 1: Implemented pipeline showing the extraction of narrative map and explainable AI components to aid users in understanding the underlying model, including topic clusters, connection labels, and storyline names.
• Figure 2: Example explanations using a COVID-19 dataset (not used in the tasks of the user study, shown only as an illustration). (a) Topical cluster explanations for the low-level space, including a scatter plot of the space and an overlay of the main storyline. Tooltips with the same color as the corresponding cluster are also included. In this figure, we also highlighted the corresponding description on the topic list. (b) Explanation for an edge between two events. This explanation specifies the type of connection (e.g., topical or entity-based), the topics of the events, and the keyword contributions that directly impact their similarity and thus the coherence score.
• Figure 3: A tally of the answers to our survey questions on the explainable AI components.