Inferential Question Answering

TL;DR

Inferential QA addresses questions whose answers are not explicitly stated in the corpus, introducing QUIT, a large-scale hint-based dataset with 7,401 questions and 2.4 million passages. The authors evaluate a full retrieval–reranking–reading pipeline and find that existing methods struggle to infer answers from indirect clues, with retrievers underperforming, rerankers offering limited gains, and even reasoning-oriented LLMs failing to surpass smaller general-purpose models. The work demonstrates that fine-tuning alone is insufficient and highlights the need for novel inference-aware retrieval and reasoning strategies. By isolating inference from direct answer containment, the paper lays groundwork for QA systems that reason over distributed evidence, aligning AI capabilities more closely with human-like comprehension.

Abstract

Despite extensive research on a wide range of question answering (QA) systems, most existing work focuses on answer containment-i.e., assuming that answers can be directly extracted and/or generated from documents in the corpus. However, some questions require inference, i.e., deriving answers that are not explicitly stated but can be inferred from the available information. We introduce Inferential QA -- a new task that challenges models to infer answers from answer-supporting passages which provide only clues. To study this problem, we construct QUIT (QUestions requiring Inference from Texts) dataset, comprising 7,401 questions and 2.4M passages built from high-convergence human- and machine-authored hints, labeled across three relevance levels using LLM-based answerability and human verification. Through comprehensive evaluation of retrievers, rerankers, and LLM-based readers, we show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements. Even reasoning-oriented LLMs fail to outperform smaller general-purpose models. These findings reveal that current QA pipelines are not yet ready for inference-based reasoning. Inferential QA thus establishes a new class of QA tasks that move towards understanding and reasoning from indirect textual evidence.

Figures (8)

• Figure 1: Example of answer-containing passages and answer-supporting passages for the question Which Spanish-speaking footballer holds the most titles?, whose answer is Lionel Messi. Green highlights indicate semantic similarity between the question and sentences (darker green = higher similarity). Bold words mark lexical overlaps between the question and passages. Blue rounded rectangles denote the relevance level of each passage to the question. In the Answer Supporting column, the entities that each passage implicitly describes with respect to the question are shown on the right side of the passages.
• Figure 2: The pipeline for constructing the Quit dataset. (1) Question Sampling: Questions (Qs) and their hints (Hs) are first filtered to avoid answer leakage. For each question, the top-5 hints are selected based on convergence, followed by question type detection and difficulty estimation. Valid questions are then sampled into temporary training, development, and test sets. (2) Dataset Preparation: For each question, subsets and permutations of hints are generated to form diverse passages (Ps). These passages are labeled automatically, with dev and test labels further verified and updated through human verification. The final corpus includes 7,401 questions and 2,405,325 passages across training, development, and test splits.
• Figure 3: Prompt of GPT-Eval. The placeholder <question> represents the question, <ground_truth> indicates the correct answer, and <candidate> shows the answer generated by different LLMs.
• Figure 4: Left: Hit@$k$ comparison across different corpora. Right: Hit@100 with respect to the number of relevant passages (label 2) across corpora.
• Figure 5: Trend of EM across different settings using DPR (vanilla and fine-tuned) as the retriever and MonoT5 (vanilla and fine-tuned) as the reranker, evaluated with LLaMA 3.2 1B, Gemma 3 4B, and Qwen 3 8B as readers in the RAG setup.