Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels

Sil Hamilton, Rebecca M. M. Hicke, Matthew Wilkens, David Mimno

TL;DR

This paper introduces the Too Long, Didn't Model (TLDM) benchmark to evaluate long-context understanding in LLMs using 40 novels spanning a wide token range. It defines three narrative tasks—chapter-wise summarization, storyworld state tracking, and narrative time estimation—and tests model performance across four window sizes and multiple text-treatment conditions. The results show that seven frontier models struggle to maintain stable long-context understanding beyond 64k tokens, with performance particularly sensitive to input order and text length, though model scale offers some gains. By releasing data, code, and a scalable evaluation framework, TLDM provides a practical, literature-grounded benchmark for assessing and improving long-context reasoning in LLMs and motivates future work on mechanistic interpretability of narrative understanding.

Abstract

Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult. We argue that novels provide a case study of subtle, complicated structure and long-range semantic dependencies often over 128k tokens in length. Inspired by work on computational novel analysis, we release the Too Long, Didn't Model (TLDM) benchmark, which tests a model's ability to report plot summary, storyworld configuration, and elapsed narrative time. We find that none of seven tested frontier LLMs retain stable understanding beyond 64k tokens. Our results suggest language model developers must look beyond "lost in the middle" benchmarks when evaluating model performance in complex long-context scenarios. To aid in further development we release the TLDM benchmark together with reference code and data.

Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels

TL;DR

This paper introduces the Too Long, Didn't Model (TLDM) benchmark to evaluate long-context understanding in LLMs using 40 novels spanning a wide token range. It defines three narrative tasks—chapter-wise summarization, storyworld state tracking, and narrative time estimation—and tests model performance across four window sizes and multiple text-treatment conditions. The results show that seven frontier models struggle to maintain stable long-context understanding beyond 64k tokens, with performance particularly sensitive to input order and text length, though model scale offers some gains. By releasing data, code, and a scalable evaluation framework, TLDM provides a practical, literature-grounded benchmark for assessing and improving long-context reasoning in LLMs and motivates future work on mechanistic interpretability of narrative understanding.

Abstract

Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult. We argue that novels provide a case study of subtle, complicated structure and long-range semantic dependencies often over 128k tokens in length. Inspired by work on computational novel analysis, we release the Too Long, Didn't Model (TLDM) benchmark, which tests a model's ability to report plot summary, storyworld configuration, and elapsed narrative time. We find that none of seven tested frontier LLMs retain stable understanding beyond 64k tokens. Our results suggest language model developers must look beyond "lost in the middle" benchmarks when evaluating model performance in complex long-context scenarios. To aid in further development we release the TLDM benchmark together with reference code and data.

Paper Structure

This paper contains 30 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. State sequencing.
  4. Long-context benchmarking.
  5. Literature as benchmark.
  6. Methods
  7. Data.
  8. Tasks.
  9. Windows of interest.
  10. Text treatments.
  11. Evaluation.
  12. Results
  13. Full-novel performance.
  14. Treatment impact.
  15. Discussion and Conclusion
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Token lengths of ten popular novels as tokenized by Gemma 3 contrasted with common maximum LLM context lengths (log) as indicated by dotted lines.
  • Figure 2: The five treatments considered in our study as applied to a six-chapter novel. Brackets indicate user message beginning and end while rectangles indicate chapters. Crosshatches indicate absence of input.
  • Figure 3: Accuracy scores per task averaged over all models. Values for each window of interest are presented. Note performance consistently degrades as input length increases.