NeoQA: Evidence-based Question Answering with Generated News Events

TL;DR

NeoQA tackles the challenge of evaluating evidence-based reasoning in large language models by constructing a self-contained, fully generated benchmark built on fictional timelines and articles to avoid pretraining data contamination. It enables controlled evaluation across evidence conditions, including sufficient, insufficient, and irrelevant evidence, and reveals that modern LLMs frequently rely on shortcut reasoning when evidence is lacking. The study demonstrates notable parametric knowledge interference in real-world benchmarks and shows mixed effects of prompting strategies and context length on multi-hop reasoning. Overall, NeoQA provides a robust framework for assessing grounded QA performance and points to directions for building more trustworthy, evidence-based AI systems.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) in large language models (LLMs) is challenging because benchmarks can quickly become stale. Questions initially requiring retrieval may become answerable from pretraining knowledge as newer models incorporate more recent information during pretraining, making it difficult to distinguish evidence-based reasoning from recall. We introduce NeoQA (News Events for Out-of-training Question Answering), a benchmark designed to address this issue. To construct NeoQA, we generated timelines and knowledge bases of fictional news events and entities along with news articles and Q\&A pairs to prevent LLMs from leveraging pretraining knowledge, ensuring that no prior evidence exists in their training data. We propose our dataset as a new platform for evaluating evidence-based question answering, as it requires LLMs to generate responses exclusively from retrieved evidence and only when sufficient evidence is available. NeoQA enables controlled evaluation across various evidence scenarios, including cases with missing or misleading details. Our findings indicate that LLMs struggle to distinguish subtle mismatches between questions and evidence, and suffer from short-cut reasoning when key information required to answer a question is missing from the evidence, underscoring key limitations in evidence-based reasoning.

Figures (19)

• Figure 1: Left:NeoQA features LLM-generated questions and documents about events from a fictional timeline, ensuring that LLMs can only answer by reasoning over the documents. Right: Real-world RAG datasets become ineffective for newer LLMs that have internalized knowledge of recent events, rendering the provided evidence documents redundant.
• Figure 2: An extract of a timeline from NeoQA with six out of ten events (summarized for visualization) with highlighted fictional named entities. Answering a multi-hop question requires combining information from two events. The model should deflect when only partial (insufficient) information is available or when subtle permutations make the question unanswerable (e.g., false premise questions).
• Figure 3: GPT-4 Turbo accuracy on RealTimeQA questions (no RAG evidence provided). It answers older questions more accurately from memory, suggesting that older RAG datasets can be solved without RAG.
• Figure 4: Events are generated sequentially based on a summary sentence and the previously generated events.
• Figure 5: Model deflection ratio in multi-hop questions with varying evidence gaps.