Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection

Kabir Ahuja, Melanie Sclar, Yulia Tsvetkov

TL;DR

The paper introduces plot hole detection as a rigorous proxy for deep narrative understanding in language models and presents FlawedFictionsMaker to controllably inject continuity errors into human-written stories, yielding the FlawedFictions benchmark. It establishes two tasks—binary plot-hole detection and localization—and analyzes frontier models, showing significant gaps, especially for longer narratives, with some models reaching near-human performance only under specific conditions. It also explores the impact of prompting strategies and reasoning budgets, and demonstrates that LLM-generated summaries and contemporary adaptations exhibit more plot holes than human-authored originals. Finally, the work provides a framework for measuring logical consistency in generated narratives and discusses future directions for training data and broader applications.

Abstract

Stories are a fundamental aspect of human experience. Engaging deeply with stories and spotting plot holes -- inconsistencies in a storyline that break the internal logic or rules of a story's world -- requires nuanced reasoning skills, including tracking entities and events and their interplay, abstract thinking, pragmatic narrative understanding, commonsense and social reasoning, and theory of mind. As Large Language Models (LLMs) increasingly generate, interpret, and modify text, rigorously assessing their narrative consistency and deeper language understanding becomes critical. However, existing benchmarks focus mainly on surface-level comprehension. In this work, we propose plot hole detection in stories as a proxy to evaluate language understanding and reasoning in LLMs. We introduce FlawedFictionsMaker, a novel algorithm to controllably and carefully synthesize plot holes in human-written stories. Using this algorithm, we construct a benchmark to evaluate LLMs' plot hole detection abilities in stories -- FlawedFictions -- , which is robust to contamination, with human filtering ensuring high quality. We find that state-of-the-art LLMs struggle in accurately solving FlawedFictions regardless of the reasoning effort allowed, with performance significantly degrading as story length increases. Finally, we show that LLM-based story summarization and story generation are prone to introducing plot holes, with more than 50% and 100% increases in plot hole detection rates with respect to human-written originals.

Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection

TL;DR

The paper introduces plot hole detection as a rigorous proxy for deep narrative understanding in language models and presents FlawedFictionsMaker to controllably inject continuity errors into human-written stories, yielding the FlawedFictions benchmark. It establishes two tasks—binary plot-hole detection and localization—and analyzes frontier models, showing significant gaps, especially for longer narratives, with some models reaching near-human performance only under specific conditions. It also explores the impact of prompting strategies and reasoning budgets, and demonstrates that LLM-generated summaries and contemporary adaptations exhibit more plot holes than human-authored originals. Finally, the work provides a framework for measuring logical consistency in generated narratives and discusses future directions for training data and broader applications.

Abstract

Stories are a fundamental aspect of human experience. Engaging deeply with stories and spotting plot holes -- inconsistencies in a storyline that break the internal logic or rules of a story's world -- requires nuanced reasoning skills, including tracking entities and events and their interplay, abstract thinking, pragmatic narrative understanding, commonsense and social reasoning, and theory of mind. As Large Language Models (LLMs) increasingly generate, interpret, and modify text, rigorously assessing their narrative consistency and deeper language understanding becomes critical. However, existing benchmarks focus mainly on surface-level comprehension. In this work, we propose plot hole detection in stories as a proxy to evaluate language understanding and reasoning in LLMs. We introduce FlawedFictionsMaker, a novel algorithm to controllably and carefully synthesize plot holes in human-written stories. Using this algorithm, we construct a benchmark to evaluate LLMs' plot hole detection abilities in stories -- FlawedFictions -- , which is robust to contamination, with human filtering ensuring high quality. We find that state-of-the-art LLMs struggle in accurately solving FlawedFictions regardless of the reasoning effort allowed, with performance significantly degrading as story length increases. Finally, we show that LLM-based story summarization and story generation are prone to introducing plot holes, with more than 50% and 100% increases in plot hole detection rates with respect to human-written originals.

Paper Structure

This paper contains 33 sections, 2 equations, 17 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Defining Plot Holes: Continuity Errors
  3. Automatically Generating Plot Holes in Stories
  4. FlawedFictions: Tasks, Metrics, and Dataset Statistics
  5. How Well do Frontier LLMs Perform on FlawedFictions?
  6. Results
  7. Measuring Logical Consistency in LLM Generated Narratives
  8. Related Work
  9. Conclusion
  10. Appendix
  11. A More Formal Treatment of Continuity Errors
  12. Human Annotation and Benchmarking
  13. Verifying stories from FlawedFictionsMaker
  14. Benchmarking Human Performance.
  15. Dataset Statistics.
  16. ...and 18 more sections

Figures (17)

  • Figure 1: Example of FlawedFictionsMaker (without the filtering step) in action that can be used to introduce plot holes in a plot hole-free story.
  • Figure 2: Continuity Error Detection Rate for stories generated using different LLMs for summarization and contemporary adaptation tasks.
  • Figure 4: An example of our human annotation interface for verifying outputs of FlawedFictionsMaker.
  • Figure 5: An example of the interface used for benchmarking human performance on FlawedFictions.
  • Figure 6: Effect of inference time compute represented using the average number of completion tokens on the performance on FlawedFictions and FlawedFictionsLong.
  • ...and 12 more figures

Theorems & Definitions (2)

  • Definition 2.1: Continuity Error
  • Definition A.1: Continuity Error with Beliefs Incorporated