Beyond Correlations: A Downstream Evaluation Framework for Query Performance Prediction

TL;DR

This work tackles the misalignment between traditional QPP evaluation (set-level correlations) and downstream utility in IR. It introduces a downstream-aware framework that treats QPP as predicting a distribution of performance across multiple rankers, and uses these predictions as priors in a weighted fusion scheme, formalized as $\theta_f(q, d; \Theta, \Phi) = \sum_{\theta \in \Theta} \phi(q, L_\theta(q))\,\sigma(q, d; \theta)$. Empirical results on TREC DL'19/20 show that QPP-guided fusion often improves AP@100 and nDCG@10 over unweighted baselines, with score-based unsupervised QPP models (e.g., NQC, RSD, UEF) typically offering the strongest gains and widening the gap with supervised approaches. The study also demonstrates a divergence between standard QPP correlations and downstream fusion performance, and documents a moderate per-query linkage between ranker-preference predictions and fusion gains, indicating potential for per-query ranker selection. Overall, the paper provides a practical, downstream-oriented perspective on QPP evaluation and demonstrates tangible retrieval benefits from QPP-informed fusion strategies, suggesting further work in adaptive, per-query ranking pipelines and RAG systems.

Abstract

The standard practice of query performance prediction (QPP) evaluation is to measure a set-level correlation between the estimated retrieval qualities and the true ones. However, neither this correlation-based evaluation measure quantifies QPP effectiveness at the level of individual queries, nor does this connect to a downstream application, meaning that QPP methods yielding high correlation values may not find a practical application in query-specific decisions in an IR pipeline. In this paper, we propose a downstream-focussed evaluation framework where a distribution of QPP estimates across a list of top-documents retrieved with several rankers is used as priors for IR fusion. While on the one hand, a distribution of these estimates closely matching that of the true retrieval qualities indicates the quality of the predictor, their usage as priors on the other hand indicates a predictor's ability to make informed choices in an IR pipeline. Our experiments firstly establish the importance of QPP estimates in weighted IR fusion, yielding substantial improvements of over 4.5% over unweighted CombSUM and RRF fusion strategies, and secondly, reveal new insights that the downstream effectiveness of QPP does not correlate well with the standard correlation-based QPP evaluation.

Figures (1)

• Figure 1: Correlation between the normalised per-query QPP effectiveness ($\tau_\Theta$) and downstream IR metric improvement ($\Delta_\mathcal{P}$, where $\mathcal{P} \in \{\text{AP@100}, \text{nDCG@10}\}$) for RSD-weighted CSUM (the best performing predictor from Table \ref{['tab:qpp_fusion_final']}) over unweighted CSUM. Pearson’s $\rho$ values are: (a) for DL'19, $\rho = 0.5695$ (AP@100) and $\rho = 0.2789$ (nDCG@10); and (b) for DL'20, $\rho = 0.2876$ (AP@100) and $\rho = 0.2307$ (nDCG@10).