Breaking Flat: A Generalised Query Performance Prediction Evaluation Framework

TL;DR

This paper generalizes query performance prediction (QPP) into a two-dimensional framework over queries and ranking models, enabling evaluation across multiple rankers and supporting mixture-of-experts-style routing. It defines three task settings—Single-Ranker Multi-Query (SRMQ-PP), Multi-Ranker Single-Query (MRSQ-PP), and Multi-Ranker Multi-Query (MRMQ-PP)—and formalizes the QPP landscape as a matrix φ ∈ $\mathbb{R}^{|Q|\times|\Theta|}$. Through extensive experiments on MS MARCO and DL datasets with eight IR models and a broad suite of QPP methods (unsupervised and supervised), the study finds that score-based QPP methods excel in SRMQ-PP, while content-based, supervised approaches (e.g., BERT-QPP, DM) better predict the best ranker for a given query (MRSQ-PP) and perform well in MRMPQ. The results show that SRMQ-PP is comparatively easier than MRSQ-PP, and the joint MRMQ measure provides a robust evaluation but with higher variance, underscoring the value of the generalized framework for robust, statistically testable comparisons. These insights have practical implications for adaptive ranking pipelines and MoE routing in IR, and the authors provide code for reproducibility and future exploration of per-query ranker selection models.

Abstract

The traditional use-case of query performance prediction (QPP) is to identify which queries perform well and which perform poorly for a given ranking model. A more fine-grained and arguably more challenging extension of this task is to determine which ranking models are most effective for a given query. In this work, we generalize the QPP task and its evaluation into three settings: (i) SingleRanker MultiQuery (SRMQ-PP), corresponding to the standard use case; (ii) MultiRanker SingleQuery (MRSQ-PP), which evaluates a QPP model's ability to select the most effective ranker for a query; and (iii) MultiRanker MultiQuery (MRMQ-PP), which considers predictions jointly across all query ranker pairs. Our results show that (a) the relative effectiveness of QPP models varies substantially across tasks (SRMQ-PP vs. MRSQ-PP), and (b) predicting the best ranker for a query is considerably more difficult than predicting the relative difficulty of queries for a given ranker.

Figures (1)

• Figure 1: A generalised QPP evaluation framework with an additional dimension of ranking models -- a): each slice represents per-ranker QPP estimates across a set of queries and their correlations with the corresponding ground-truth retrieval quality (Equation \ref{['eq:standard']}, \ref{['eq:standard_ext']}); b): each slice represents per-query predictions of rankers' performance and their correlations with the corresponding ground-truth retrieval quality for each IR model (see Equation \ref{['eq:local']}); c): computes a correlation between $|\Theta| |Q|=nm$ predictions and retrieval quality for each query–ranker pair (Equation \ref{['eq:global']}).