Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Locations of Characters in Narratives: Andersen and Persuasion Datasets

Batuhan Ozyurt, Roya Arkhmammadova, Deniz Yuret

TL;DR

This work introduces two narrative datasets, Andersen and Persuasion, with manually annotated character-location pairs to evaluate narrative spatial understanding in AI. Through prompt-based QA evaluations of multiple LLMs, the study finds that best accuracies are modest (around 62% for Andersen and 56% for Persuasion), highlighting the difficulty of reliably locating characters within complex texts. The paper also analyzes robustness to prompt distractions and the impact of in-context learning, observing that distractions degrade performance while in-context gains are context-dependent. Together, the datasets and evaluation framework advance computational narrative understanding and spatial reasoning research by providing concrete benchmarks and analysis of prompting strategies.

Abstract

The ability of machines to grasp spatial understanding within narrative contexts is an intriguing aspect of reading comprehension that continues to be studied. Motivated by the goal to test the AI's competence in understanding the relationship between characters and their respective locations in narratives, we introduce two new datasets: Andersen and Persuasion. For the Andersen dataset, we selected fifteen children's stories from "Andersen's Fairy Tales" by Hans Christian Andersen and manually annotated the characters and their respective locations throughout each story. Similarly, for the Persuasion dataset, characters and their locations in the novel "Persuasion" by Jane Austen were also manually annotated. We used these datasets to prompt Large Language Models (LLMs). The prompts are created by extracting excerpts from the stories or the novel and combining them with a question asking the location of a character mentioned in that excerpt. Out of the five LLMs we tested, the best-performing one for the Andersen dataset accurately identified the location in 61.85% of the examples, while for the Persuasion dataset, the best-performing one did so in 56.06% of the cases.

Locations of Characters in Narratives: Andersen and Persuasion Datasets

TL;DR

This work introduces two narrative datasets, Andersen and Persuasion, with manually annotated character-location pairs to evaluate narrative spatial understanding in AI. Through prompt-based QA evaluations of multiple LLMs, the study finds that best accuracies are modest (around 62% for Andersen and 56% for Persuasion), highlighting the difficulty of reliably locating characters within complex texts. The paper also analyzes robustness to prompt distractions and the impact of in-context learning, observing that distractions degrade performance while in-context gains are context-dependent. Together, the datasets and evaluation framework advance computational narrative understanding and spatial reasoning research by providing concrete benchmarks and analysis of prompting strategies.

Abstract

The ability of machines to grasp spatial understanding within narrative contexts is an intriguing aspect of reading comprehension that continues to be studied. Motivated by the goal to test the AI's competence in understanding the relationship between characters and their respective locations in narratives, we introduce two new datasets: Andersen and Persuasion. For the Andersen dataset, we selected fifteen children's stories from "Andersen's Fairy Tales" by Hans Christian Andersen and manually annotated the characters and their respective locations throughout each story. Similarly, for the Persuasion dataset, characters and their locations in the novel "Persuasion" by Jane Austen were also manually annotated. We used these datasets to prompt Large Language Models (LLMs). The prompts are created by extracting excerpts from the stories or the novel and combining them with a question asking the location of a character mentioned in that excerpt. Out of the five LLMs we tested, the best-performing one for the Andersen dataset accurately identified the location in 61.85% of the examples, while for the Persuasion dataset, the best-performing one did so in 56.06% of the cases.

Paper Structure

This paper contains 16 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Information
  4. Discussion on the Differences between the Datasets
  5. Inter-annotator Agreement
  6. Evaluation Metrics
  7. Baseline Measurements
  8. Prompting Large Language Models
  9. Adding Distractions to the Textual Prompt
  10. In-Context Learning
  11. Conclusion
  12. Bibliographical References
  13. Annotation Guideline
  14. Prompt Templates
  15. Examples For In-Context Learning
  16. ...and 1 more sections

Figures (3)

  • Figure 1: A prompt template for question answering.
  • Figure 2: A snippet from "The Swineherd", a story from the Andersen dataset. Here, at the end of the snippet, the location of the Princess is "by the pigsty", but it is not explicitly stated.
  • Figure 3: 2-shot in-context learning prompt example