LitVISTA: A Benchmark for Narrative Orchestration in Literary Text

TL;DR

LitVISTA introduces VISTA Space, a higher-dimensional framework that unifies human and model narrative representations, and LitVISTA, a structurally annotated benchmark grounded in literary texts, to evaluate narrative orchestration. It shifts analysis from flat sequences to topologies defined by Impulse, Resonance, and Pause anchors, enabling systematic evaluation of global narrative structure. Oracle evaluations across frontier LLMs reveal persistent gaps in capturing unified narrative views, with reasoning-enabled prompting offering limited, model-specific gains. Overall, LitVISTA provides a precise benchmark and a computational model of narrative dynamics to advance computational narratology.

Abstract

Computational narrative analysis aims to capture rhythm, tension, and emotional dynamics in literary texts. Existing large language models can generate long stories but overly focus on causal coherence, neglecting the complex story arcs and orchestration inherent in human narratives. This creates a structural misalignment between model- and human-generated narratives. We propose VISTA Space, a high-dimensional representational framework for narrative orchestration that unifies human and model narrative perspectives. We further introduce LitVISTA, a structurally annotated benchmark grounded in literary texts, enabling systematic evaluation of models' narrative orchestration capabilities. We conduct oracle evaluations on a diverse selection of frontier LLMs, including GPT, Claude, Grok, and Gemini. Results reveal systematic deficiencies: existing models fail to construct a unified global narrative view, struggling to jointly capture narrative function and structure. Furthermore, even advanced thinking modes yield only limited gains for such literary narrative understanding.

Figures (6)

• Figure 1: Comparison of story arcs between human and LLM storytellers. This image, reproduced from tian2024large, shows that LLM-generated stories often have simpler arcs and earlier turning points, whereas human-authored narratives are more complex.
• Figure 2: VISTA Space and its projections.. The center illustrates VISTA Space, a higher-dimensional representation of narrative orchestration. The surrounding panels show three projections: the human picture of narrative experience (left), the LLM picture based on token-level representations (bottom-right), and the VISTA-induced picture (top-right), which situates human and model representations within a unified structural perspective.
• Figure 3: The process begins with LitBank text data. Experts A and B independently annotate Verb$^+$ roles in Phase 1. In Phase 2, dependency parsing is conducted by Experts C and D. Phase 3 resolves any conflicts through adjudication, producing the final LitVISTA graph.
• Figure 4: Oracle evaluation results. The scatter plot shows Anchor F1 (x-axis) versus Dependency F1 (y-axis) for each model.
• Figure 5: Frequency of narrative dependencies by absolute character offset distance. The X-axis represents distance buckets, and the Y-axis shows different dependency types.