Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias

TL;DR

This work examines how fixed top-k retrieval in retrieval-augmented generation introduces distractors and suffers from a lost-in-the-middle effect. It introduces a dynamic context selection framework that combines a per-query k classifier with an LLM-based reranker to adaptively choose the number and relevance of retrieved passages. Through extensive experiments on MuSiQue-Ans and related datasets, the approach demonstrates that dynamic retrieval, especially when paired with reranking and thoughtful context placement, yields meaningful improvements in exact-match and F1 scores for multi-hop QA tasks. The results highlight the importance of input structure and adaptive retrieval in mitigating noise and maximizing useful context for generation.

Abstract

Retrieval Augmented Generation (RAG) enhances language model performance by incorporating external knowledge retrieved from large corpora, which makes it highly suitable for tasks such as open domain question answering. Standard RAG systems typically rely on a fixed top k retrieval strategy, which can either miss relevant information or introduce semantically irrelevant passages, known as distractors, that degrade output quality. Additionally, the positioning of retrieved passages within the input context can influence the model attention and generation outcomes. Context placed in the middle tends to be overlooked, which is an issue known as the "lost in the middle" phenomenon. In this work, we systematically analyze the impact of distractors on generation quality, and quantify their effects under varying conditions. We also investigate how the position of relevant passages within the context window affects their influence on generation. Building on these insights, we propose a context-size classifier that dynamically predicts the optimal number of documents to retrieve based on query-specific informational needs. We integrate this approach into a full RAG pipeline, and demonstrate improved performance over fixed k baselines.

Figures (2)

• Figure 1: Evaluation of the impact of additional context for 2- to 4-hop and including the presence of distractors (d).
• Figure 2: Diagram of Baseline Pipeline, Classifier-k Pipeline, and Classifier-k+LLM pipeline