Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Query Variants: The Advantage of Retrieval Over Generation of Query Variants for Effective QPP

Fangzheng Tian, Debasis Ganguly, Craig Macdonald

TL;DR

The paper tackles the problem of predicting query performance for neural IR models by replacing or augmenting generated query variants with retrieved QVs from a large training set. It introduces a two-step neighbourhood expansion (1-hop and 2-hop) to retrieve high-recall QVs and uses a re-ranked, similarity-weighted aggregation to yield improved QPP signals. Empirically, retrieved QVs consistently outperform generated QVs, with 2-hop QVs offering additional gains, notably achieving up to about 20% improvements on neural ranking models like MonoT5. The approach demonstrates practical value for QPP in real-world systems and suggests future work integrating retrieval-based QVs with LLM-driven QV generation to further reduce topical drift. Overall, the work provides a robust, retrieval-centered framework to enhance QPP for neural IR, with clear guidance on hyperparameter sensitivity and evaluation on standard benchmarks.

Abstract

Leveraging query variants (QVs), i.e., queries with potentially similar information needs to the target query, has been shown to improve the effectiveness of query performance prediction (QPP) approaches. Existing QV-based QPP methods generate QVs facilitated by either query expansion or non-contextual embeddings, which may introduce topical drifts and hallucinations. In this paper, we propose a method that retrieves QVs from a training set (e.g., MS MARCO) for a given target query of QPP. To achieve a high recall in retrieving queries with the most similar information needs as the target query from a training set, we extend the directly retrieved QVs (1-hop QVs) by a second retrieval using their denoted relevant documents (which yields 2-hop QVs). Our experiments, conducted on TREC DL'19 and DL'20, show that the QPP methods with QVs retrieved by our method outperform the best-performing existing generated-QV-based QPP approaches by as much as around 20\%, on neural ranking models like MonoT5.

Revisiting Query Variants: The Advantage of Retrieval Over Generation of Query Variants for Effective QPP

TL;DR

The paper tackles the problem of predicting query performance for neural IR models by replacing or augmenting generated query variants with retrieved QVs from a large training set. It introduces a two-step neighbourhood expansion (1-hop and 2-hop) to retrieve high-recall QVs and uses a re-ranked, similarity-weighted aggregation to yield improved QPP signals. Empirically, retrieved QVs consistently outperform generated QVs, with 2-hop QVs offering additional gains, notably achieving up to about 20% improvements on neural ranking models like MonoT5. The approach demonstrates practical value for QPP in real-world systems and suggests future work integrating retrieval-based QVs with LLM-driven QV generation to further reduce topical drift. Overall, the work provides a robust, retrieval-centered framework to enhance QPP for neural IR, with clear guidance on hyperparameter sensitivity and evaluation on standard benchmarks.

Abstract

Leveraging query variants (QVs), i.e., queries with potentially similar information needs to the target query, has been shown to improve the effectiveness of query performance prediction (QPP) approaches. Existing QV-based QPP methods generate QVs facilitated by either query expansion or non-contextual embeddings, which may introduce topical drifts and hallucinations. In this paper, we propose a method that retrieves QVs from a training set (e.g., MS MARCO) for a given target query of QPP. To achieve a high recall in retrieving queries with the most similar information needs as the target query from a training set, we extend the directly retrieved QVs (1-hop QVs) by a second retrieval using their denoted relevant documents (which yields 2-hop QVs). Our experiments, conducted on TREC DL'19 and DL'20, show that the QPP methods with QVs retrieved by our method outperform the best-performing existing generated-QV-based QPP approaches by as much as around 20\%, on neural ranking models like MonoT5.

Paper Structure

This paper contains 27 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unsupervised QPP methods.
  4. QV-based QPP methods.
  5. Research Gaps.
  6. Query Variant Retrieval based QPP Methodology
  7. Smoothing Estimates from QVs
  8. Retrieving the Query Variants
  9. 1-hop QV retrieval.
  10. 2-hop QV retrieval.
  11. Evaluation
  12. Research Questions
  13. Experiment Settings
  14. Datasets.
  15. Target Retrievers.
  16. ...and 12 more sections

Figures (4)

  • Figure 1: An visualisation of the idea behind QPP with retrieved QVs. For a target query (the blue point), the first step is to retrieve a list of 1-hop QVs (the green points) from an index constructed from the training query set. In the second step, the relevant document associated with each 1-hop candidate (the italicised texts) is used as a query to further retrieve 2-hop QVs (the amber points). The combination of 1-hop and 2-hop QVs comprises the target query's 2-hop neighbourhood, which defines the range of the final candidate QV set.
  • Figure 2: Schematic representation of the idea of collecting second-hop neighbourhood queries by extending the set of 1-hop neighbours $\pazocal{E}^1(Q)$ with the set of relevant documents $R(Q')$ for each first-hop query $Q'$.
  • Figure 3: A complete spectrum of the results of Table \ref{['table:ret_qv_main']} showing the effect of $k$ for each tested QPP method. The base predictor is NQC.
  • Figure 4: Effect of jointly varying both $k$ (#QVs), and $\lambda$ (relative importance of the QVs) on DL'19 and DL'20 test queries. The base estimator is NQC, and the target IR metric is AP@100.