Analyzing the Effectiveness of Listwise Reranking with Positional Invariance on Temporal Generalizability

TL;DR

This paper investigates how listwise reranking, particularly ListT5 with Fusion-in-Decoder, performs under temporal distribution shifts in a real-world dynamic setting. By leveraging the LongEval benchmark, the study shows that listwise reranking can significantly improve temporal generalizability and mitigate positional bias, with pronounced gains as temporal drift increases (notably on the test-long subset). The authors compare pointwise and listwise approaches and demonstrate that ListT5 outperforms MonoT5 across metrics in zero-shot, drifted scenarios, supporting the viability of robust, time-aware ranking in dynamic information environments. The work emphasizes practical deployment implications, including efficient architectures (FiD-based ListT5), transparent evaluation using a proxy metric, and careful data preprocessing to reflect real-world noisy corpora.

Abstract

This working note outlines our participation in the retrieval task at CLEF 2024. We highlight the considerable gap between studying retrieval performance on static knowledge documents and understanding performance in real-world environments. Therefore, Addressing these discrepancies and measuring the temporal persistence of IR systems is crucial. By investigating the LongEval benchmark, specifically designed for such dynamic environments, our findings demonstrate the effectiveness of a listwise reranking approach, which proficiently handles inaccuracies induced by temporal distribution shifts. Among listwise rerankers, our findings show that ListT5, which effectively mitigates the positional bias problem by adopting the Fusion-in-Decoder architecture, is especially effective, and more so, as temporal drift increases, on the test-long subset.

Figures (4)

• Figure 1: Overview of the longeval retrieval challenge. The task is evaluated into two parts: short-term persistency and long-term persistency.
• Figure 2: Explanation of listwise reranking models with respect to the pointwise ranking variants. Pointwise reranking individually assigns relevance scores to each documents, where listwise reranking feeds a list of documents at once to the model and let the model generate the relative order of documents.
• Figure 3: (Figure borrowed from listt5) Illustration of different sorting strategies for listwise reranking, mainly the sliding window approach used for listwise reranking models for with LLMs and the tournament sort approach used for ListT5. In the example, the number of total candidate passages $n$ is 8, window size is 4, and stride is 2 where the hyperparameter $r$ for ListT5 is 2. Please refer to the ListT5 paper listt5 for more detailed explanation about tournament sort.
• Figure 4: Overview of the retrieval process of the submission to the LongEval challenge.