Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Prediction ROC Bands for Graph Classification

Yujia Wu, Bo Yang, Elynn Chen, Yuzhou Chen, Zheshi Zheng

TL;DR

Conditional Prediction ROC (CP-ROC) bands are introduced, offering uncertainty quantification for ROC curves and robustness to distributional shifts in test data, which enhances uncertainty quantification efficiency and reliability for ROC curves, proving valuable for real-world applications with non-iid objects.

Abstract

Graph classification in medical imaging and drug discovery requires accuracy and robust uncertainty quantification. To address this need, we introduce Conditional Prediction ROC (CP-ROC) bands, offering uncertainty quantification for ROC curves and robustness to distributional shifts in test data. Although developed for Tensorized Graph Neural Networks (TGNNs), CP-ROC is adaptable to general Graph Neural Networks (GNNs) and other machine learning models. We establish statistically guaranteed coverage for CP-ROC under a local exchangeability condition. This addresses uncertainty challenges for ROC curves under non-iid setting, ensuring reliability when test graph distributions differ from training data. Empirically, to establish local exchangeability for TGNNs, we introduce a data-driven approach to construct local calibration sets for graphs. Comprehensive evaluations show that CP-ROC significantly improves prediction reliability across diverse tasks. This method enhances uncertainty quantification efficiency and reliability for ROC curves, proving valuable for real-world applications with non-iid objects.

Conditional Prediction ROC Bands for Graph Classification

TL;DR

Conditional Prediction ROC (CP-ROC) bands are introduced, offering uncertainty quantification for ROC curves and robustness to distributional shifts in test data, which enhances uncertainty quantification efficiency and reliability for ROC curves, proving valuable for real-world applications with non-iid objects.

Abstract

Graph classification in medical imaging and drug discovery requires accuracy and robust uncertainty quantification. To address this need, we introduce Conditional Prediction ROC (CP-ROC) bands, offering uncertainty quantification for ROC curves and robustness to distributional shifts in test data. Although developed for Tensorized Graph Neural Networks (TGNNs), CP-ROC is adaptable to general Graph Neural Networks (GNNs) and other machine learning models. We establish statistically guaranteed coverage for CP-ROC under a local exchangeability condition. This addresses uncertainty challenges for ROC curves under non-iid setting, ensuring reliability when test graph distributions differ from training data. Empirically, to establish local exchangeability for TGNNs, we introduce a data-driven approach to construct local calibration sets for graphs. Comprehensive evaluations show that CP-ROC significantly improves prediction reliability across diverse tasks. This method enhances uncertainty quantification efficiency and reliability for ROC curves, proving valuable for real-world applications with non-iid objects.

Paper Structure

This paper contains 22 sections, 1 theorem, 28 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setups
  4. Conditional Prediction ROC Bands
  5. Soft Conformal Prediction
  6. CP-ROC Bands for Exchangeable Data
  7. CP-ROC Bands for Non-Exchangeable Data
  8. Extension to Multi-label Classification
  9. Conditional CP-ROC Bands for Tensorized Graph Neural Networks
  10. Theoretical Gaurantees
  11. Experiments
  12. Experiment Settings
  13. Results
  14. Extension to Regression Task
  15. Conclusion
  16. ...and 7 more sections

Key Result

Theorem 1

Assume $\mathcal{G}_{test}$ are i.i.d data set, and each $\mathcal{G}_j\in\mathcal{G}_{test}$ is i.i.d with its $K$-nearest calibration sets $\mathcal{G}_{calib}^j$ and also $K$-nearest training sets $\mathcal{G}_{train}^j$. Let $F_{jk}(\cdot)$ denotes the CDF of $\{s_i\}_{i:\mathcal{G}_i\in\mathcal In addition, we assume $F_{jk}^{-1}(\alpha/2)<0<F_{jk}^{-1}(1-\alpha/2)$. Then for any $\lambda\in(

Figures (6)

  • Figure 1: ROC bands for TPR (Top) and FPR (Bottom).
  • Figure 2: Example of exchangeable ROC bands for TGNN and GIN.
  • Figure 3: Performance comparison among three regression models and varied dataset size. ( top) fixed specificity and ( bottom) fixed sensitivity confidence bands.
  • Figure 4: Comparison between Bootstrap and our UQ on PROTEIN dataset
  • Figure 5: Exchangeable ROC bands of TGNN and GIN comparison.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 1