Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TorchCP: A Python Library for Conformal Prediction

Jianguo Huang, Jianqing Song, Xuanning Zhou, Bingyi Jing, Hongxin Wei

TL;DR

TorchCP addresses the need to scale conformal prediction to cutting-edge deep learning by providing a PyTorch-native toolbox that integrates CP algorithms with DNNs, GNNs, and LLMs. Its low-coupling three-module architecture (Trainer, Score, Predictor), GPU-accelerated batch processing, and broad algorithmic coverage (6 training methods, 17 scores, 10 predictors) enable efficient, flexible uncertainty quantification for large-scale models. Empirical results show substantial inference-time improvements (up to 90% faster than baselines on large datasets) and competitive predictive coverage across vision, graph, and language tasks, with extensive tutorials and documentation. The library also emphasizes extensibility and multi-model support, positioning TorchCP as a scalable, open-source platform for robust uncertainty quantification in modern AI applications.

Abstract

Conformal prediction (CP) is a powerful statistical framework that generates prediction intervals or sets with guaranteed coverage probability. While CP algorithms have evolved beyond traditional classifiers and regressors to sophisticated deep learning models like deep neural networks (DNNs), graph neural networks (GNNs), and large language models (LLMs), existing CP libraries often lack the model support and scalability for large-scale deep learning (DL) scenarios. This paper introduces TorchCP, a PyTorch-native library designed to integrate state-of-the-art CP algorithms into DL techniques, including DNN-based classifiers/regressors, GNNs, and LLMs. Released under the LGPL-3.0 license, TorchCP comprises about 16k lines of code, validated with 100\% unit test coverage and detailed documentation. Notably, TorchCP enables CP-specific training algorithms, online prediction, and GPU-accelerated batch processing, achieving up to 90\% reduction in inference time on large datasets. With its low-coupling design, comprehensive suite of advanced methods, and full GPU scalability, TorchCP empowers researchers and practitioners to enhance uncertainty quantification across cutting-edge applications.

TorchCP: A Python Library for Conformal Prediction

TL;DR

TorchCP addresses the need to scale conformal prediction to cutting-edge deep learning by providing a PyTorch-native toolbox that integrates CP algorithms with DNNs, GNNs, and LLMs. Its low-coupling three-module architecture (Trainer, Score, Predictor), GPU-accelerated batch processing, and broad algorithmic coverage (6 training methods, 17 scores, 10 predictors) enable efficient, flexible uncertainty quantification for large-scale models. Empirical results show substantial inference-time improvements (up to 90% faster than baselines on large datasets) and competitive predictive coverage across vision, graph, and language tasks, with extensive tutorials and documentation. The library also emphasizes extensibility and multi-model support, positioning TorchCP as a scalable, open-source platform for robust uncertainty quantification in modern AI applications.

Abstract

Conformal prediction (CP) is a powerful statistical framework that generates prediction intervals or sets with guaranteed coverage probability. While CP algorithms have evolved beyond traditional classifiers and regressors to sophisticated deep learning models like deep neural networks (DNNs), graph neural networks (GNNs), and large language models (LLMs), existing CP libraries often lack the model support and scalability for large-scale deep learning (DL) scenarios. This paper introduces TorchCP, a PyTorch-native library designed to integrate state-of-the-art CP algorithms into DL techniques, including DNN-based classifiers/regressors, GNNs, and LLMs. Released under the LGPL-3.0 license, TorchCP comprises about 16k lines of code, validated with 100\% unit test coverage and detailed documentation. Notably, TorchCP enables CP-specific training algorithms, online prediction, and GPU-accelerated batch processing, achieving up to 90\% reduction in inference time on large datasets. With its low-coupling design, comprehensive suite of advanced methods, and full GPU scalability, TorchCP empowers researchers and practitioners to enhance uncertainty quantification across cutting-edge applications.
Paper Structure (34 sections, 11 equations, 8 figures, 5 tables)

This paper contains 34 sections, 11 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Main contributions
  3. Advantages for DL scenarios
  4. Methodological scope
  5. Batch Processing and GPU Support
  6. Tutorials
  7. Related library
  8. Conclusion and Future Work
  9. Details of Conformal Prediction Methods in TorchCP
  10. Split CP
  11. DNN Classifier
  12. DNN Regressor
  13. GNN
  14. LLM
  15. Metrics
  16. ...and 19 more sections

Figures (8)

  • Figure 1: The modules of TorchCP.
  • Figure 2: Results of image classification with $\alpha=0.1$ on ImageNet dataset.
  • Figure 3: Conditional coverage results on ImageNet image classification with $\alpha = 0.1$.
  • Figure 4: Results of regression with $\alpha=0.1$ on Community dataset.
  • Figure 5: Results of Graph node classification with $\alpha=0.1$ on CoraML dataset.
  • ...and 3 more figures