Evaluation of Temporal Change in IR Test Collections

TL;DR

This work addresses the under-explored temporal dimension of information retrieval evaluations by formalizing evolving evaluation environments (EE) through a CRUD-based DTQ taxonomy (Documents, Topics, Qrels under Create, Update, Delete). It adapts reproducibility and time-aware measures, including $RBO$, $RMSE$, $\mathcal{R}_e\Delta$, and $\Delta\mathrm{RI}$, to quantify how retrieval effectiveness changes across time and under different change scenarios. Using three dynamic test collections—TripClick, TREC-COVID, and LongEval—and five state-of-the-art systems, the study shows that both average effectiveness and system rankings are highly dependent on the specific temporal scenario and dataset, with ranking similarity often diminishing over time. The findings highlight the importance of incorporating temporal dynamics into IR evaluations to assess robustness, reusability of test collections, and fair comparison across evolving environments, and they call for broader temporal test collections and per-topic analyses to better capture user-facing impacts.

Abstract

Information retrieval systems have been evaluated using the Cranfield paradigm for many years. This paradigm allows a systematic, fair, and reproducible evaluation of different retrieval methods in fixed experimental environments. However, real-world retrieval systems must cope with dynamic environments and temporal changes that affect the document collection, topical trends, and the individual user's perception of what is considered relevant. Yet, the temporal dimension in IR evaluations is still understudied. To this end, this work investigates how the temporal generalizability of effectiveness evaluations can be assessed. As a conceptual model, we generalize Cranfield-type experiments to the temporal context by classifying the change in the essential components according to the create, update, and delete operations of persistent storage known from CRUD. From the different types of change different evaluation scenarios are derived and it is outlined what they imply. Based on these scenarios, renowned state-of-the-art retrieval systems are tested and it is investigated how the retrieval effectiveness changes on different levels of granularity. We show that the proposed measures can be well adapted to describe the changes in the retrieval results. The experiments conducted confirm that the retrieval effectiveness strongly depends on the evaluation scenario investigated. We find that not only the average retrieval performance of single systems but also the relative system performance are strongly affected by the components that change and to what extent these components changed.

Figures (2)

• Figure 1: Overview of the shared concepts between the test collections that stem from different evolving environments.
• Figure 2: Retrieval effectiveness of the different systems measured by bpref for the three test collections over multiple sub-collections. The effectiveness is strongly influenced by the changes in the sub-collection. No strong agreement for a system ranking between test collections or sub-collections can be found.