LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA

TL;DR

We address the reliability of NarrativeQA for long-document narrative QA by introducing LiteraryQA, a high-quality, human- and LLM-validated subset focused on literary works. We implement a two-phase data refinement pipeline (document-level filtering/cleaning and QA-level refinement) and conduct a comprehensive meta-evaluation of evaluation metrics, including LLM-based judging. Our findings show that n-gram metrics correlate poorly with human judgments on NarrativeQA, though METEOR performs comparatively better on LiteraryQA, while LLM-as-a-judge—especially in the summary-based setting—achieves the strongest alignment with human judgments (up to around $r\approx 0.68$). We benchmark long-context LLMs on LiteraryQA, demonstrating the dataset’s utility for evaluating narrative understanding and guiding future benchmark design, while noting remaining challenges and avenues for future work such as reduced reliance on LLM validation and exploration of retrieval-augmented strategies.

Abstract

Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of NarrativeQA, the most widely used benchmark in this domain, is hindered by noisy documents and flawed QA pairs. In this work, we introduce LiteraryQA, a high-quality subset of NarrativeQA focused on literary works. Using a human- and LLM-validated pipeline, we identify and correct low-quality QA samples while removing extraneous text from source documents. We then carry out a meta-evaluation of automatic metrics to clarify how systems should be evaluated on LiteraryQA. This analysis reveals that all n-gram-based metrics have a low system-level correlation to human judgment, while LLM-as-a-Judge evaluations, even with small open-weight models, can strongly agree with the ranking identified by humans. Finally, we benchmark a set of long-context LLMs on LiteraryQA. We release our code and data at https://github.com/SapienzaNLP/LiteraryQA.

Figures (11)

• Figure 1: Illustrative example of the failure modes that automatic metrics incur when evaluating predictions on the original NarrativeQA.
• Figure 2: Standardized metrics of Claude 3.5 Haiku's generated responses grouped by the absolute length difference between the prediction and the reference answers. Each bin contains an equal number of samples.
• Figure 3: Prometheus-as-a-judge scores of the models across different settings.
• Figure 4: Comparison of document lengths between NarrativeQA and LiteraryQA for the same books. Blue line indicates absolute token differences; red line shows percentage differences.
• Figure 5: List of the most represented authors in LiteraryQA (test set).