Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distribution-informed Efficient Conformal Prediction for Full Ranking

Wenbo Liao, Huipeng Huang, Chen Jia, Huajun Xi, Hao Zeng, Hongxin Wei

TL;DR

This work tackles uncertainty quantification for full ranking by introducing Distribution-informed Conformal Ranking (DCR), which leverages the exact Negative Hypergeometric rank distribution to derive the latent non-conformity score distribution and construct efficient, valid prediction sets for absolute test ranks. By forming a Mixture CDF over conditional score distributions, DCR avoids the conservatism of prior envelope-based methods (e.g., TCPR) and provably achieves smaller prediction sets at a given miscoverage level; it also provides finite-sample FCP concentration guarantees. To scale to large problems, the authors propose MDCR, a Monte-Carlo variant with $O(n \\log n)$ complexity that preserves marginal coverage while significantly reducing computation. Empirical results across multiple datasets and ranking models show that DCR reduces average prediction-set size (up to ~36% relative to TCPR) while maintaining valid coverage, and VA-based scoring improves local adaptivity. Overall, DCR advances practical, distribution-free uncertainty quantification for full ranking with a scalable, theoretically grounded framework.

Abstract

Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications. Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items based on the relative ranks of calibration items. However, relying on upper bounds of non-conformity scores renders the method overly conservative, resulting in substantially large prediction sets. To address this, we propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores. In particular, we find that the absolute ranks of calibration items follow Negative Hypergeometric distributions, conditional on their relative ranks. DCR thus uses the rank distribution to derive non-conformity score distribution and determine conformal thresholds. We provide theoretical guarantees that DCR achieves improved efficiency over the baseline while ensuring valid coverage under mild assumptions. Extensive experiments demonstrate the superiority of DCR, reducing average prediction set size by up to 36%, while maintaining valid coverage.

Distribution-informed Efficient Conformal Prediction for Full Ranking

TL;DR

This work tackles uncertainty quantification for full ranking by introducing Distribution-informed Conformal Ranking (DCR), which leverages the exact Negative Hypergeometric rank distribution to derive the latent non-conformity score distribution and construct efficient, valid prediction sets for absolute test ranks. By forming a Mixture CDF over conditional score distributions, DCR avoids the conservatism of prior envelope-based methods (e.g., TCPR) and provably achieves smaller prediction sets at a given miscoverage level; it also provides finite-sample FCP concentration guarantees. To scale to large problems, the authors propose MDCR, a Monte-Carlo variant with complexity that preserves marginal coverage while significantly reducing computation. Empirical results across multiple datasets and ranking models show that DCR reduces average prediction-set size (up to ~36% relative to TCPR) while maintaining valid coverage, and VA-based scoring improves local adaptivity. Overall, DCR advances practical, distribution-free uncertainty quantification for full ranking with a scalable, theoretically grounded framework.

Abstract

Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications. Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items based on the relative ranks of calibration items. However, relying on upper bounds of non-conformity scores renders the method overly conservative, resulting in substantially large prediction sets. To address this, we propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores. In particular, we find that the absolute ranks of calibration items follow Negative Hypergeometric distributions, conditional on their relative ranks. DCR thus uses the rank distribution to derive non-conformity score distribution and determine conformal thresholds. We provide theoretical guarantees that DCR achieves improved efficiency over the baseline while ensuring valid coverage under mild assumptions. Extensive experiments demonstrate the superiority of DCR, reducing average prediction set size by up to 36%, while maintaining valid coverage.
Paper Structure (58 sections, 12 theorems, 59 equations, 8 figures, 2 tables, 5 algorithms)

This paper contains 58 sections, 12 theorems, 59 equations, 8 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Full ranking setup.
  4. Transductive conformal prediction for ranking (TCPR).
  5. Methodology and theoretical analysis
  6. Probabilistic modeling of unobserved ranks.
  7. Deriving the latent score distribution.
  8. Distribution-informed conformal ranking (DCR).
  9. Efficiency and comparison with TCPR.
  10. Finite-Sample FCP concentration.
  11. Scalable stochastic approximation of DCR.
  12. Experiments
  13. Experimental setup
  14. Datasets and models.
  15. Baselines and evaluation metrics.
  16. ...and 43 more sections

Key Result

Theorem 2.3

If Assumption ass:exchange holds and the rank bounds $\{(R_i^-,R_i^+)\}_{i\in[n]}$ satisfy eq:tcpr-rank-bounds, then for any test item $X_{n+j}$ , the prediction set defined in eq:tcpr-pred-set satisfies

Figures (8)

  • Figure 1: Prediction set size of TCPR and oracle CP at different target error levels. The experiment is conducted with RankNet 10.1145/1102351.1102363 on a synthetic dataset. For TCPR, rank bounds are constructed using the quantile envelope, as it provides the tightest bound. Since both methods achieve the desired coverage, only the prediction set size is presented.
  • Figure 2: Empirical coverage and prediction efficiency of RankNet under different target miscoverage levels $\alpha$ on the Synthetic dataset. The results are averaged over 1,000 independent trials with $n=100$ and $m=500$. The shaded areas represent standard deviation.
  • Figure 3: Runtime comparison of DCR, MDCR, and TCPR on Yummly28k using LambdaMART. MDCR is the most computationally efficient.
  • Figure 4: Finite-sample FCP convergence of RankNet on the Synthetic dataset with respect to sample sizes. Results are averaged over 1,000 trials.
  • Figure 5: Visualization of prediction intervals on the Anime dataset. The scatter plots display the True Rank versus the Predicted Rank for LambdaMART. The curves represent the prediction sets constructed by DCR-RA (orange) and DCR-VA (green) at $\alpha=0.1$.
  • ...and 3 more figures

Theorems & Definitions (25)

  • Remark 2.2: Handling Ties
  • Theorem 2.3
  • Proposition 3.1: Conditional Rank Distribution
  • Theorem 3.2: Marginal Validity
  • Theorem 3.3: Efficiency Guarantee
  • Remark 3.4: Analysis of Conservatism vs. Oracle
  • Theorem 3.5: Finite-Sample FCP Concentration
  • Remark 3.6: Asymptotic Validity
  • Theorem 3.7: Marginal Validity of MDCR
  • Theorem 3.8: Asymptotic Variance Comparison
  • ...and 15 more