Transductive Conformal Inference for Full Ranking

TL;DR

The paper tackles the problem of quantifying uncertainty in full ranking when only the relative order of a subset of items is known. It introduces a transductive conformal prediction approach that leverages bounds on unknown conformity scores to produce marginally valid prediction sets for the ranks of all items and to control the false coverage proportion across multiple predictions. Two score paradigms, RA and VA, are developed, with theoretical envelopes and numerical Monte Carlo envelopes to bound calibration-to-test rank transfers, significantly reducing prediction-set length compared with naive bounds. Empirical results on synthetic data and real-world datasets (Yummly-10k and Anime LTR) demonstrate robust FCP control and competitive, adaptive interval lengths across state-of-the-art ranking algorithms such as RankNet and LambdaMART.

Abstract

We introduce a method based on Conformal Prediction (CP) to quantify the uncertainty of full ranking algorithms. We focus on a specific scenario where $n+m$ items are to be ranked by some ``black box'' algorithm. It is assumed that the relative (ground truth) ranking of $n$ of them is known. The objective is then to quantify the error made by the algorithm on the ranks of the $m$ new items among the total $(n+m)$. In such a setting, the true ranks of the $n$ original items in the total $(n+m)$ depend on the (unknown) true ranks of the $m$ new ones. Consequently, we have no direct access to a calibration set to apply a classical CP method. To address this challenge, we propose to construct distribution-free bounds of the unknown conformity scores using recent results on the distribution of conformal p-values. Using these scores upper bounds, we provide valid prediction sets for the rank of any item. We also control the false coverage proportion, a crucial quantity when dealing with multiple prediction sets. Finally, we empirically show on both synthetic and real data the efficiency of our CP method for state-of-the-art algorithms such as RankNet or LambdaMart.

Figures (13)

• Figure 1: The two envelopes for $n=50$, $m=500$, $\delta = 0.1$ and $K= 10^5$. The blue and red lines are respectively the linear and quantile envelopes, and each black curve is an ordered realization of $R^{c+t}$. The green line is the envelope from Prop. \ref{['lemma:bound_R^t_j']}.
• Figure 2: Synthetic data: FCP and relative lengths obtained for RankNet with the ( RA) and ( VA) score, for the quantile (Qenv) and linear (Lenv) envelopes when $m=500$ and $n \in \IfEqCase{a}{{a}{\mathopen{}\mathclose{\left\{100, 500, 2500\right\}}}{0}{\{100, 500, 2500\}}{1}{\{100, 500, 2500\}}{2}{\{100, 500, 2500\}}{3}{\{100, 500, 2500\}}{4}{\{100, 500, 2500\}}}[]$. White circles represent the means.
• Figure 3: Data from Section \ref{['sec:diff_ra_va']}: True ranks $R^{c+t}_{n+j}$ in function of their predicted rank ${\widehat{R}}^{c+t}_{n+j}$ by RankNet and their prediction sets with scores $s^{{\bf RA}\xspace}$ and $s^{{\bf VA}\xspace}$ for $n=m=500$.
• Figure 4: Ranks predicted by LambdaMART(leaves, trees) in function of the ranks predicted by a larger model of $400$ trees of $20$ leaves.
• Figure 5: Synthetic data: True ranks $R^{c+t}_{n+j}$ in function of their predicted rank ${\widehat{R}}^{c+t}_{n+j}$ by RankNet and their prediction sets with scores $s^{{\bf RA}\xspace}$ and $s^{{\bf VA}\xspace}$ for $n=m=500$.

Theorems & Definitions (16)

• Theorem 2.1: vovk2005algorithmiclei2018distribution
• Remark 3.3
• Remark 3.4
• Theorem 3.5
• Example 3.6: $s= s^{{\bf RA}\xspace}$
• Example 3.7: $s =s^{{\bf VA}\xspace}$
• Proposition 3.8
• Proposition 4.1
• Proposition 4.2
• Theorem A.1