Validating Search Query Simulations: A Taxonomy of Measures
Andreas Konstantin Kruff, Nolwenn Bernard, Philipp Schaer
TL;DR
The paper addresses the challenge of validating user query simulators for evaluating information retrieval systems, noting the lack of standardized methodologies. It conducts a comprehensive literature review to build a taxonomy of validation facets and measures, then empirically corroborates the taxonomy by analyzing relationships among measures across four public datasets using exploratory factor analysis, Pearson $\rho$ and Kendall $\tau$ correlations, and normalized mutual information (NMI). A dedicated Python library is released to compute the measures and support one-to-one and one-to-many comparisons, enhancing reproducibility. Key findings show traditional IR metrics are highly redundant, while query similarity measures and SERP overlap provide complementary insights, suggesting that combined, multi-facet validation yields a more robust assessment of simulated queries.
Abstract
Assessing the validity of user simulators when used for the evaluation of information retrieval systems remains an open question, constraining their effective use and the reliability of simulation-based results. To address this issue, we conduct a comprehensive literature review with a particular focus on methods for the validation of simulated user queries with regard to real queries. Based on the review, we develop a taxonomy that structures the current landscape of available measures. We empirically corroborate the taxonomy by analyzing the relationships between the different measures applied to four different datasets representing diverse search scenarios. Finally, we provide concrete recommendations on which measures or combinations of measures should be considered when validating user simulation in different contexts. Furthermore, we release a dedicated library with the most commonly used measures to facilitate future research.