FaNS: a Facet-based Narrative Similarity Metric

TL;DR

Experimental results demonstrate that the FaNS metric exhibits a higher correlation than traditional text similarity metrics that directly measure the lexical/semantic match between narratives, demonstrating its effectiveness in comparing the finer details between a pair of narratives.

Abstract

Similar Narrative Retrieval is a crucial task since narratives are essential for explaining and understanding events, and multiple related narratives often help to create a holistic view of the event of interest. To accurately identify semantically similar narratives, this paper proposes a novel narrative similarity metric called Facet-based Narrative Similarity (FaNS), based on the classic 5W1H facets (Who, What, When, Where, Why, and How), which are extracted by leveraging the state-of-the-art Large Language Models (LLMs). Unlike existing similarity metrics that only focus on overall lexical/semantic match, FaNS provides a more granular matching along six different facets independently and then combines them. To evaluate FaNS, we created a comprehensive dataset by collecting narratives from AllSides, a third-party news portal. Experimental results demonstrate that the FaNS metric exhibits a higher correlation (37\% higher) than traditional text similarity metrics that directly measure the lexical/semantic match between narratives, demonstrating its effectiveness in comparing the finer details between a pair of narratives.

Figures (5)

• Figure 1: Computing Facet-based Narrative Similarity (FaNS) with Large Language Models (LLMs)
• Figure 2: Various levels of prompts used to retrieve 5W1H facets
• Figure 3: A visual illustration of numerous topic groups found in the Allsides data
• Figure 4: Kendall ($\tau$) correlation coefficient when alpha ($\alpha)$$\in [0, 1]$ for Level 3, Level 2, and Level 1 prompting to retrieve 5W1H facets using ChatGPT. Here, alpha ($\alpha)$ denotes weight for entity-specific facets (who, when, where), and (1- $\alpha$) weight is given to descriptive facets (what, why, how).
• Figure 5: Kendall ($\tau$) Correlation coefficient when alpha ($\alpha)$$\in [0, 1]$ for Level 3, Level 2, and Level 1 prompting to retrieve 5W1H facets using Google Bard. Here, alpha ($\alpha)$ denotes weight for entity specifics (who, when, where) and (1- $\alpha$) weight is given to descriptive specifics (what, why, how).