LLM-Assisted Pseudo-Relevance Feedback

TL;DR

To address topic drift in pseudo-relevance feedback, the paper proposes filtering the initial top-ranked documents with an LLM before performing RM3 expansion. It introduces two variants: a straightforward LLM-based filter and a second variant that also leverages the LLM's next-token probability to weight the RM3 relevance model. Across AP85-89, ROBUST04, and MSMARCO, the LLM-filtered RM3 variants consistently outperform RM3 and strong reranking baselines, and prompting with narrative sections further boosts performance and reduces damaged queries. The approach preserves interpretability, grounds expansions in actual corpus evidence, and incurs only modest overhead, suggesting practical gains for real-world IR pipelines.

Abstract

Query expansion is a long-standing technique to mitigate vocabulary mismatch in ad hoc Information Retrieval. Pseudo-relevance feedback methods, such as RM3, estimate an expanded query model from the top-ranked documents, but remain vulnerable to topic drift when early results include noisy or tangential content. Recent approaches instead prompt Large Language Models to generate synthetic expansions or query variants. While effective, these methods risk hallucinations and misalignment with collection-specific terminology. We propose a hybrid alternative that preserves the robustness and interpretability of classical PRF while leveraging LLM semantic judgement. Our method inserts an LLM-based filtering stage prior to RM3 estimation: the LLM judges the documents in the initial top-$k$ ranking, and RM3 is computed only over those accepted as relevant. This simple intervention improves over blind PRF and a strong baseline across several datasets and metrics.