Recall, Robustness, and Lexicographic Evaluation
Fernando Diaz, Michael D. Ekstrand, Bhaskar Mitra
TL;DR
This work formalizes recall in rankings via recall-orientation, robustness, and lexicographic evaluation, introducing Total Search Efficiency (TSE) as a principled recall-focused metric and Lexicographic Recall (LR) to address ties and improve discriminative power. It connects recall to robustness by showing that the worst-case user or provider performance corresponds to TSE, aligning with Rawlsian fairness concepts. The empirical analysis across 3 recommendation tasks and 17 information retrieval tasks demonstrates that LR preserves correlation with existing recall metrics while offering greater sensitivity and stability under missing labels. The findings advocate adopting recall-oriented, robustness-aware evaluation (via TSE and LR) to improve fairness, reliability, and interpretability in ranking systems. The approach provides practical guidance for data labeling, depth of evaluation, and potential algorithmic directions toward stochastic ranking mechanisms. The work thus deepens the theoretical and empirical understanding of recall and its links to robustness and fairness in modern information access systems.
Abstract
Although originally developed to evaluate sets of items, recall is often used to evaluate rankings of items, including those produced by recommender, retrieval, and other machine learning systems. The application of recall without a formal evaluative motivation has led to criticism of recall as a vague or inappropriate measure. In light of this debate, we reflect on the measurement of recall in rankings from a formal perspective. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation. First, we formally define `recall-orientation' as the sensitivity of a metric to a user interested in finding every relevant item. Second, we analyze recall-orientation from the perspective of robustness with respect to possible content consumers and providers, connecting recall to recent conversations about fair ranking. Finally, we extend this conceptual and theoretical treatment of recall by developing a practical preference-based evaluation method based on lexicographic comparison. Through extensive empirical analysis across three recommendation tasks and 17 information retrieval tasks, we establish that our new evaluation method, lexirecall, has convergent validity (i.e., it is correlated with existing recall metrics) and exhibits substantially higher sensitivity in terms of discriminative power and stability in the presence of missing labels. Our conceptual, theoretical, and empirical analysis substantially deepens our understanding of recall and motivates its adoption through connections to robustness and fairness.