Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning An Interpretable Risk Scoring System for Maximizing Decision Net Benefit

Wenhao Chi, Ş. İlker Birbil

Abstract

Risk scoring systems are widely used in high-stakes domains to assist decision-making. However, existing approaches often focus on optimizing predictive accuracy or likelihood-based criteria, which may not align with the main goal of maximizing utility. In this paper, we propose a novel risk scoring system that directly optimizes net benefit over a range of decision thresholds. The model is formulated as a sparse integer linear programming problem which enables the construction of a transparent scoring system with integer coefficients, and hence, facilitates interpretation and practical application. We also establish fundamental relationships among net benefit, discrimination, and calibration. Our analysis proves that optimizing net benefit also guarantees conventional performance measures. We thoroughly evaluated our method on multiple public datasets as well as on a real-world clinical dataset. This computational study demonstrated that our interpretable method can effectively achieve high net benefit while maintaining competitive discrimination and calibration performance.

Learning An Interpretable Risk Scoring System for Maximizing Decision Net Benefit

Abstract

Risk scoring systems are widely used in high-stakes domains to assist decision-making. However, existing approaches often focus on optimizing predictive accuracy or likelihood-based criteria, which may not align with the main goal of maximizing utility. In this paper, we propose a novel risk scoring system that directly optimizes net benefit over a range of decision thresholds. The model is formulated as a sparse integer linear programming problem which enables the construction of a transparent scoring system with integer coefficients, and hence, facilitates interpretation and practical application. We also establish fundamental relationships among net benefit, discrimination, and calibration. Our analysis proves that optimizing net benefit also guarantees conventional performance measures. We thoroughly evaluated our method on multiple public datasets as well as on a real-world clinical dataset. This computational study demonstrated that our interpretable method can effectively achieve high net benefit while maintaining competitive discrimination and calibration performance.

Paper Structure

This paper contains 19 sections, 7 theorems, 60 equations, 5 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Risk Scoring Systems.
  3. Decision Curve Analysis.
  4. Method
  5. Illustrative Application to a Clinical Dataset
  6. Discrimination and Utility
  7. Calibration and Utility
  8. Integer Programming Formulation
  9. Learning Capacity and Generalization
  10. Experiments
  11. Experimental setup
  12. Results
  13. Case Study: Invasiveness of Lung Adenocarcinoma
  14. Data Description
  15. Experimental setup
  16. ...and 4 more sections

Key Result

Theorem 1

For given thresholds $0=p_0 < p_i < \cdots < p_{M+1}=1$, let $P_k=\sum_{i=1}^k \frac{(p_{i+1}-p_i)p_i}{1-p_i}$, $k=1,2,\cdots,M$, then where $a_0=\frac{N^+}{N}$, and $A_k(\cdot;a_0)$ is a function defined in $[0,1]$, satisfying Moreover, these bounds are tight. $\blacktriangleleft$$\blacktriangleleft$

Figures (5)

  • Figure 1: Relationship between AUROC and AUNBC. (a) The theoretical relationship, generated with $M=4$, $a_0=0.5$, and $p_i=i/5$ for $i=0,1,\cdots,4$; (b) The relationship under the setting of \ref{['subsec:method_example']} with RSS-DNB model and two types of synthetic predictions.
  • Figure 2: AUROC-AUNBC pairs for the random synthetic predictions in \ref{['subsec:method_example']} before and after applying \ref{['alg:improve_aunbc']}. Arrows indicate the direction of improvement. The solid curve represents the theoretical boundary and the circle denotes the proposed model.
  • Figure 3: ROC curves for each dataset. Each panel shows the ROC curves of all models evaluated on the corresponding dataset.
  • Figure 4: Calibration plots for each dataset. Each panel shows the calibration plots of all models evaluated on the corresponding dataset.
  • Figure 5: Decision curves for each dataset. Each panel shows the decision curves of all models evaluated on the corresponding dataset.

Theorems & Definitions (25)

  • Definition 1
  • Definition 2
  • Remark 1
  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Theorem 2
  • proof
  • Remark 2
  • ...and 15 more