Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bounded-Abstention Pairwise Learning to Rank

Antonio Ferrara, Andrea Pugnana, Francesco Bonchi, Salvatore Ruggieri

TL;DR

This work introduces BALToR, a principled, bounded-abstention framework for pairwise learning-to-rank with ties. By thresholding the ranker’s conditional risk at a $c$-quantile level, BALToR selects which pairs to predict and which to abstain, targeting a fixed abstention budget. The authors provide a theoretical characterization of the optimal abstention policy, a model-agnostic plug-in algorithm, and extensive experiments on Web-30k, OHSUMED, and MQ2007 showing improved accuracy on non-abstained pairs while satisfying the coverage constraint and maintaining balanced abstention across outcome classes. The approach offers a practical, scalable way to integrate human input or higher-quality data into ranking systems, particularly in high-stakes settings where uncertainty must be carefully managed.

Abstract

Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts. This makes the integration of safety mechanisms essential. One such mechanism is $\textit{abstention}$, which enables algorithmic decision-making system to defer uncertain or low-confidence decisions to human experts. While abstention have been predominantly explored in the context of classification tasks, its application to other machine learning paradigms remains underexplored. In this paper, we introduce a novel method for abstention in pairwise learning-to-rank tasks. Our approach is based on thresholding the ranker's conditional risk: the system abstains from making a decision when the estimated risk exceeds a predefined threshold. Our contributions are threefold: a theoretical characterization of the optimal abstention strategy, a model-agnostic, plug-in algorithm for constructing abstaining ranking models, and a comprehensive empirical evaluations across multiple datasets, demonstrating the effectiveness of our approach.

Bounded-Abstention Pairwise Learning to Rank

TL;DR

This work introduces BALToR, a principled, bounded-abstention framework for pairwise learning-to-rank with ties. By thresholding the ranker’s conditional risk at a -quantile level, BALToR selects which pairs to predict and which to abstain, targeting a fixed abstention budget. The authors provide a theoretical characterization of the optimal abstention policy, a model-agnostic plug-in algorithm, and extensive experiments on Web-30k, OHSUMED, and MQ2007 showing improved accuracy on non-abstained pairs while satisfying the coverage constraint and maintaining balanced abstention across outcome classes. The approach offers a practical, scalable way to integrate human input or higher-quality data into ranking systems, particularly in high-stakes settings where uncertainty must be carefully managed.

Abstract

Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts. This makes the integration of safety mechanisms essential. One such mechanism is , which enables algorithmic decision-making system to defer uncertain or low-confidence decisions to human experts. While abstention have been predominantly explored in the context of classification tasks, its application to other machine learning paradigms remains underexplored. In this paper, we introduce a novel method for abstention in pairwise learning-to-rank tasks. Our approach is based on thresholding the ranker's conditional risk: the system abstains from making a decision when the estimated risk exceeds a predefined threshold. Our contributions are threefold: a theoretical characterization of the optimal abstention strategy, a model-agnostic, plug-in algorithm for constructing abstaining ranking models, and a comprehensive empirical evaluations across multiple datasets, demonstrating the effectiveness of our approach.

Paper Structure

This paper contains 24 sections, 3 theorems, 44 equations, 16 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Pairwise learning to rank with ties
  4. Abstaining Machine Learning
  5. Bounded Abstention Learning To Rank
  6. Experiments
  7. Experimental Settings
  8. Datasets
  9. Baselines
  10. Metrics
  11. Setup
  12. Reproducibility
  13. Experimental Results
  14. Q1: improving accuracy
  15. Q2: respecting coverage
  16. ...and 9 more sections

Key Result

Theorem 3.1

Let us consider a symmetric loss. A selection function $g^*:\mathcal{X}\times \mathcal{X}\to [0,1]$ is an optimal solution to Eq. eq:l2rank_abst if and only if the following conditions hold: where $\beta = \inf \left\{a: \iint_{r(\mathcal{X}\times\mathcal{X})_{<a}} p(\mathbf{x},\mathbf{x}')d\mathbf{x} d\mathbf{x}' \geq c \right\}$.

Figures (16)

  • Figure 1: Illustration of the role of the abstainer in a ranking process. A standard ranker produces a complete ordering of items for a given query. However, it may exhibit low confidence in the relative ordering of certain item pairs. An abstainer identifies a number of these low-confidence pairs and defer them to an evaluator for further inspection. Such an evaluator - a human or even another model - leverages additional knowledge or higher-quality data to resolve the uncertainty on those pairs and, finally, producing a more reliable final ranking.
  • Figure 2: Accuracy $Acc$ on the selected pairs ($mean\pm std$ over five folds) for the BT model (top line) and the TM model (bottom line). For smaller target coverages $c$ (i.e., more abstention), the accuracy increases for BALToR, remains stable for the random abstainer, has erratic performance for the entropy-based abstainer.
  • Figure 3: Actual (empirical) coverage $Cov$ on the test set ($mean\pm std$ over five folds) for the BT model . The actual coverage remains very close to the target coverage $c$.
  • Figure 4: Distribution of classes $SelRate$ on the selected pairs ($mean\pm std$ over five folds) for the BT model . While the target coverage $c$ varies, BALToR maintains stable the proportions of the classes in $\mathcal{Y} = \{-1,0,1\}$.
  • Figure 5: Estimated density functions over MQ2007 Fold 1 calibration set for BT (\ref{['fig:BTMQ2007_xgb_entropy']}) and TM (\ref{['fig:TMMQ2007_xgb_entropy']}) when using an XGBRanker. The shapes of the density functions differ (are similar) when using BT (TM) model.
  • ...and 11 more figures

Theorems & Definitions (6)

  • Theorem 3.1
  • Theorem 3.2
  • Proposition 3.3
  • proof
  • proof
  • proof