Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Risk Control for Ordinal Classification

Yunpeng Xu, Wenge Guo, Zhi Wei

TL;DR

The paper addresses uncertainty quantification for ordinal classification by embedding the task in the conformal risk control in expectation framework and deriving risk bounds. It proposes two loss formulations, weight-based and divergence-based, and develops corresponding algorithms to construct prediction sets that bound the expected loss at a specified level $α$; both Oracle and Marginal variants are provided. Theoretical results include risk bounds and a generalized lower bound under a relaxed discontinuity assumption, with experiments on simulated data, UTKFace age groups, and diabetic retinopathy indicating effective risk control and characteristic behavior differences between the two loss types. The findings demonstrate that prediction sets can be tuned to reflect class importance or penalize large errors, offering practical guidance for ordinal problems with safety or fairness considerations, and point to future work on conditional coverage and weight selection.

Abstract

As a natural extension to the standard conformal prediction method, several conformal risk control methods have been recently developed and applied to various learning problems. In this work, we seek to control the conformal risk in expectation for ordinal classification tasks, which have broad applications to many real problems. For this purpose, we firstly formulated the ordinal classification task in the conformal risk control framework, and provided theoretic risk bounds of the risk control method. Then we proposed two types of loss functions specially designed for ordinal classification tasks, and developed corresponding algorithms to determine the prediction set for each case to control their risks at a desired level. We demonstrated the effectiveness of our proposed methods, and analyzed the difference between the two types of risks on three different datasets, including a simulated dataset, the UTKFace dataset and the diabetic retinopathy detection dataset.

Conformal Risk Control for Ordinal Classification

TL;DR

The paper addresses uncertainty quantification for ordinal classification by embedding the task in the conformal risk control in expectation framework and deriving risk bounds. It proposes two loss formulations, weight-based and divergence-based, and develops corresponding algorithms to construct prediction sets that bound the expected loss at a specified level ; both Oracle and Marginal variants are provided. Theoretical results include risk bounds and a generalized lower bound under a relaxed discontinuity assumption, with experiments on simulated data, UTKFace age groups, and diabetic retinopathy indicating effective risk control and characteristic behavior differences between the two loss types. The findings demonstrate that prediction sets can be tuned to reflect class importance or penalize large errors, offering practical guidance for ordinal problems with safety or fairness considerations, and point to future work on conditional coverage and weight selection.

Abstract

As a natural extension to the standard conformal prediction method, several conformal risk control methods have been recently developed and applied to various learning problems. In this work, we seek to control the conformal risk in expectation for ordinal classification tasks, which have broad applications to many real problems. For this purpose, we firstly formulated the ordinal classification task in the conformal risk control framework, and provided theoretic risk bounds of the risk control method. Then we proposed two types of loss functions specially designed for ordinal classification tasks, and developed corresponding algorithms to determine the prediction set for each case to control their risks at a desired level. We demonstrated the effectiveness of our proposed methods, and analyzed the difference between the two types of risks on three different datasets, including a simulated dataset, the UTKFace dataset and the diabetic retinopathy detection dataset.
Paper Structure (17 sections, 3 theorems, 20 equations, 8 figures, 3 tables, 3 algorithms)

This paper contains 17 sections, 3 theorems, 20 equations, 8 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Problem Formulation
  5. Risk control
  6. Weight-based Risk
  7. An Oracle Method
  8. Marginal Risk Control
  9. Divergence-based risk
  10. An Oracle Method
  11. Marginal Risk Control
  12. Experiments
  13. Simulated Data
  14. Age Recognition
  15. Diabetic Retinopathy Detection
  16. ...and 2 more sections

Key Result

Theorem 1

Let $\hat{\lambda}$ be the value defined in equation (eq7), under the assumptions stated as above, we have $E[L(Y_{test}, C_{\hat{\lambda}}(X_{test})] \leq \alpha$. Specifically, in the settings of Theorem 4 in the Supplementary Materials, we have $\alpha - \frac{(M+2)B}{n+1} \leq E[L(Y_{test}, C_{\

Figures (8)

  • Figure 1: A simulated 10-class ordinal data
  • Figure 2: Prediction set sizes at different values of $\alpha$ on simulated data
  • Figure 3: Risk distributions of different scenarios at a fixed $\alpha$ on the simulated dataset.
  • Figure 4: Distributions of prediction centroids on simulated data.
  • Figure 5: Image examples in the UTKFace dataset.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Theorem 1
  • proof
  • Remark 1
  • Remark 2
  • Remark 3
  • Theorem 2
  • proof
  • Theorem 3