Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can we predict QPP? An approach based on multivariate outliers

Adrian-Gabriel Chifu, Sébastien Déjean, Moncef Garouani, Josiane Mothe, Diégo Ortiz, Md Zia Ullah

TL;DR

The paper tackles the limited accuracy of Query Performance Prediction (QPP) by examining predictability itself. It applies multivariate outlier detection based on Transformed Rank Correlations (TRC) to identify queries for which QPP tends to fail, using a multivariate distance approach with a 0.95 F-distribution threshold and multiple predictors. The study demonstrates that hard-to-predict queries exist and that removing them yields higher correlations between predicted and actual performance across datasets and QPPs, indicating robustness of the approach beyond a single predictor. This work proposes a new research direction: measuring and potentially abstaining from QPP when uncertainty is high, and extending the analysis to more datasets and feature-combination models.

Abstract

Query performance prediction (QPP) aims to forecast the effectiveness of a search engine across a range of queries and documents. While state-of-the-art predictors offer a certain level of precision, their accuracy is not flawless. Prior research has recognized the challenges inherent in QPP but often lacks a thorough qualitative analysis. In this paper, we delve into QPP by examining the factors that influence the predictability of query performance accuracy. We propose the working hypothesis that while some queries are readily predictable, others present significant challenges. By focusing on outliers, we aim to identify the queries that are particularly challenging to predict. To this end, we employ multivariate outlier detection method. Our results demonstrate the effectiveness of this approach in identifying queries on which QPP do not perform well, yielding less reliable predictions. Moreover, we provide evidence that excluding these hard-to-predict queries from the analysis significantly enhances the overall accuracy of QPP.

Can we predict QPP? An approach based on multivariate outliers

TL;DR

The paper tackles the limited accuracy of Query Performance Prediction (QPP) by examining predictability itself. It applies multivariate outlier detection based on Transformed Rank Correlations (TRC) to identify queries for which QPP tends to fail, using a multivariate distance approach with a 0.95 F-distribution threshold and multiple predictors. The study demonstrates that hard-to-predict queries exist and that removing them yields higher correlations between predicted and actual performance across datasets and QPPs, indicating robustness of the approach beyond a single predictor. This work proposes a new research direction: measuring and potentially abstaining from QPP when uncertainty is high, and extending the analysis to more datasets and feature-combination models.

Abstract

Query performance prediction (QPP) aims to forecast the effectiveness of a search engine across a range of queries and documents. While state-of-the-art predictors offer a certain level of precision, their accuracy is not flawless. Prior research has recognized the challenges inherent in QPP but often lacks a thorough qualitative analysis. In this paper, we delve into QPP by examining the factors that influence the predictability of query performance accuracy. We propose the working hypothesis that while some queries are readily predictable, others present significant challenges. By focusing on outliers, we aim to identify the queries that are particularly challenging to predict. To this end, we employ multivariate outlier detection method. Our results demonstrate the effectiveness of this approach in identifying queries on which QPP do not perform well, yielding less reliable predictions. Moreover, we provide evidence that excluding these hard-to-predict queries from the analysis significantly enhances the overall accuracy of QPP.
Paper Structure (5 sections, 1 equation, 3 figures, 1 table)

This paper contains 5 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Identification of difficult to predict queries by TRC
  3. Data
  4. Results
  5. Conclusion

Figures (3)

  • Figure 1: QPPs behave differently on the same system and set of queries (TREC78). Predicted AP (X-axis) and actual AP (Y-axis) obtained with LGD weighting function and their Pearson correlation. Left side In_expC2 ($\rho$=0.484) - Right side LemurTF_IDF QPP ($\rho$=0.552).
  • Figure 2: Outlier queries are different according to the QPP. Queries detected as outliers by our method are in red. The four plots correspond to the four QPPs. The X-axis is not meaningful. Y-axis corresponds to the predicted values by the considered Letor QPP. QQP values are calculated for the TREC78 collection.
  • Figure 3: Outlier queries are correctly identified. Queries that have been detected as outliers by our method are displayed in red. X-axis represents the QPP values and Y-axis is the actual AP. Here the values are calculated for LemurTF_IDF on the TREC78 collection and a model based on the LGD weighting function. 18 out of 100 queries were identified as outliers. The red line is the regression when all the topics are considered while the black one is after the outliers were removed.