Counterfactual Query Rewriting to Use Historical Relevance Feedback

TL;DR

This work addresses the problem of using historical relevance feedback when the corpus evolves and previously relevant documents may no longer exist or have changed content. It introduces counterfactual query rewriting with three approaches—boosting, relevance feedback expansion, and keyqueries—that leverage observations (q, d, t) from past relevance judgments. The methods are evaluated on the LongEval evolving web corpus, where keyqueries provide the strongest gains and the counterfactual approaches generalize to unseen content, often outperforming transformer-based baselines while remaining efficient. The findings demonstrate robust retrieval improvements under corpus dynamics and offer practical strategies for leveraging limited historical feedback in dynamic information retrieval settings.

Abstract

When a retrieval system receives a query it has encountered before, previous relevance feedback, such as clicks or explicit judgments can help to improve retrieval results. However, the content of a previously relevant document may have changed, or the document might not be available anymore. Despite this evolved corpus, we counterfactually use these previously relevant documents as relevance signals. In this paper we proposed approaches to rewrite user queries and compare them against a system that directly uses the previous qrels for the ranking. We expand queries with terms extracted from the previously relevant documents or derive so-called keyqueries that rank the previously relevant documents to the top of the current corpus. Our evaluation in the CLEF LongEval scenario shows that rewriting queries with historical relevance feedback improves the retrieval effectiveness and even outperforms computationally expensive transformer-based approaches.

Figures (2)

• Figure 1: Frequency of overlapping queries over the different timestamps.
• Figure 2: S$_{3}$ Similarities of documents with overlapping URLs as eCDF plot.