Retrieval Augmented Question Answering: When Should LLMs Admit Ignorance?

TL;DR

This work addresses the noise problem in retrieval-augmented QA caused by long-context prompts by introducing a sliding-window, divide-and-conquer prompting strategy. The approach retrieves top-$K$ Wikipedia pages with BM25 and processes them in sequential windows of size $w$, allowing the model to answer when sufficient information is present and abstain otherwise. Experiments across three open-domain QA datasets show the adaptive prompting matches or surpasses standard full-context prompting while using about 1.5x fewer tokens, though LLMs still tend to hallucinate when information is insufficient, underscoring the need for robust abstention training. Overall, the study demonstrates a more efficient and noise-resilient RAG workflow and motivates future training objectives that penalize unsupported generation and promote correct refusal.

Abstract

The success of expanded context windows in Large Language Models (LLMs) has driven increased use of broader context in retrieval-augmented generation. We investigate the use of LLMs for retrieval augmented question answering. While longer contexts make it easier to incorporate targeted knowledge, they introduce more irrelevant information that hinders the model's generation process and degrades its performance. To address the issue, we design an adaptive prompting strategy which involves splitting the retrieved information into smaller chunks and sequentially prompting a LLM to answer the question using each chunk. Adjusting the chunk size allows a trade-off between incorporating relevant information and reducing irrelevant information. Experimental results on three open-domain question answering datasets demonstrate that the adaptive strategy matches the performance of standard prompting while using fewer tokens. Our analysis reveals that when encountering insufficient information, the LLM often generates incorrect answers instead of declining to respond, which constitutes a major source of error. This finding highlights the need for further research into enhancing LLMs' ability to effectively decline requests when faced with inadequate information.

Figures (7)

• Figure 1: The pipeline of two prompting strategies. Given a question, we first retrieve some relevant pages using an off-the shelf retriever (the upper part). In the standard prompting (the lower left part), all retrieved pages along with the given question and a customized instruction, are fed to an LLM to generate the answer. In the adaptive prompting (illustrated in the lower right), we employ a sliding window approach. This window traverses the pages in descending order of retrieval score, sequentially feeding specific segments to the LLM to generate the answer.
• Figure 2: An example of zero-shot prompting
• Figure 3: An example of positive window contain $k$ Wikipedia pages. The green rectangle represents a relevant Wikipedia page with respect to a given question.
• Figure 4: An example of negative window contain $k$ Wikipedia pages. It denotes that all $k$ Wikipedia pages are irrelevant to the question and the LLM should output "answer not found" when encountering such a window.
• Figure 5: Impact of the window sliding direction. The left figure illustrates the changes in EM across various window sizes when using forward and backward sliding directions. The right figure displays the corresponding average number of negative windows for each scenario.