Exploring the Nexus Between Retrievability and Query Generation Strategies

TL;DR

The paper addresses retrievability bias and reproducibility concerns by comparing retrievability scores derived from artificial query generation with those from real user AOL logs across multiple collections. It formalizes retrievability via $r(d)=\sum_{q\in\mathsf{Q}} o_q\cdot f(k_{dq},c)$ and uses the Lorenz curve and Gini coefficient $G$ to quantify inequality, illustrating how query generation shapes bias. Empirically, five query-generation techniques yield markedly different retrievability distributions, with real AOL queries producing the largest disparities and a rule-based simulated method (RSQ) yielding the highest cross-method correlation and lowest inequality. The findings underscore the need for standardized query-generation criteria to improve reproducibility in retrievability studies and point toward rule-based, discriminative generation as a promising direction when query logs are unavailable.

Abstract

Quantifying bias in retrieval functions through document retrievability scores is vital for assessing recall-oriented retrieval systems. However, many studies investigating retrieval model bias lack validation of their query generation methods as accurate representations of retrievability for real users and their queries. This limitation results from the absence of established criteria for query generation in retrievability assessments. Typically, researchers resort to using frequent collocations from document corpora when no query log is available. In this study, we address the issue of reproducibility and seek to validate query generation methods by comparing retrievability scores generated from artificially generated queries to those derived from query logs. Our findings demonstrate a minimal or negligible correlation between retrievability scores from artificial queries and those from query logs. This suggests that artificially generated queries may not accurately reflect retrievability scores as derived from query logs. We further explore alternative query generation techniques, uncovering a variation that exhibits the highest correlation. This alternative approach holds promise for improving reproducibility when query logs are unavailable.

Figures (3)

• Figure 1: A sample Lorenz Curve. The values belonging to the distribution in which the disparity is to be observed are arranged in a sorted manner along the X-axis.
• Figure 2: Gini coefficient for each dataset when the retrievability scores are calculated using different query sets as discussed in Section \ref{['sec:exp']}.
• Figure 3: Lorenz curve with retrievability computed with various query sets on different collections.