coverforest: Conformal Predictions with Random Forest in Python

TL;DR

coverforest introduces a lightweight Python package that combines optimized random forests with modern conformal prediction techniques to produce distribution-free prediction sets and intervals. By implementing CV+, Jackknife+-after-Bootstrap, and split conformal methods (with APS and RAPS scores) and leveraging parallelism and Cython for cross-conformity computations, it achieves guaranteed coverage with substantial speedups over existing implementations. The authors provide a scikit-learn–style API, empirical demonstrations on classification and regression benchmarks, and a thorough comparison with crepes and MAPIE, highlighting both performance gains and practical usability. While CPU-bound and multi-core–oriented, the work lays out clear paths for future GPU and distributed extensions to broaden scalability and applicability in large-scale settings.

Abstract

Conformal prediction provides a framework for uncertainty quantification, specifically in the forms of prediction intervals and sets with distribution-free guaranteed coverage. While recent cross-conformal techniques such as CV+ and Jackknife+-after-bootstrap achieve better data efficiency than traditional split conformal methods, they incur substantial computational costs due to required pairwise comparisons between training and test samples' out-of-bag scores. Observing that these methods naturally extend from ensemble models, particularly random forests, we leverage existing optimized random forest implementations to enable efficient cross-conformal predictions. We present coverforest, a Python package that implements efficient conformal prediction methods specifically optimized for random forests. coverforest supports both regression and classification tasks through various conformal prediction methods, including split conformal, CV+, Jackknife+-after-bootstrap, and adaptive prediction sets. Our package leverages parallel computing and Cython optimizations to speed up out-of-bag calculations. Our experiments demonstrate that coverforest's predictions achieve the desired level of coverage. In addition, its training and prediction times can be faster than an existing implementation by 2--9 times. The source code for the coverforest is hosted on GitHub at https://github.com/donlap/coverforest.

Figures (6)

• Figure 1: Class structure and the metrics in coverforest.
• Figure 2: Coverage probabilities of CoverForestClassifier's set predictions on the test set, repeated 50 times for each of the CV+, J+ab and split conformal methods with the APS and RAPS scores. The results of crepes's methods are also included.
• Figure 3: Average size of CoverForestClassifier's set predictions on the test set, repeated 50 times for each of the CV+, J+ab and split conformal methods with the APS and RAPS scores. The results of crepes's methods are also included.
• Figure 4: Training and prediction times of coverforest's and MAPIE's implementations of the CV+, J+ab and split conformal methods with the APS score and $\alpha=0.05$. Note that MAPIE does not implement the J+ab method for classification.
• Figure 5: Coverage and average length of CoverForestRegressor's interval predictions on the test set, repeated 50 times for each of the CV+, J+ab and split conformal methods with the residuals as the conformity scores. Results of crepes's J+ab method are also included.