Calibrated Conformal Prediction Intervals for Microphysical Process Rates

Abstract

Conformal prediction can yield statistically valid prediction intervals for any regression model, with no model modifications and small computational costs. To assess its practical value, we apply conformal methods to quantify uncertainty in machine learning emulators of six microphysical process rates. Microphysical process rates describe small-scale processes in atmospheric clouds such as precipitation formation and aerosol-cloud interactions, and help understand weather and climate. The emulators are trained on simulation output from the ICOsahedral Nonhydrostatic (ICON) model in a limited-area numerical weather prediction configuration. We compare split conformal prediction for deterministic emulators with conformalized quantile regression for quantile regression emulators. Both conformal prediction methods yield well-calibrated and sharp prediction intervals on average, but conformalized quantile regression provides more consistent intervals across several orders of magnitude, making it preferable for the uncertainty quantification of climate variables.

Figures (10)

• Figure 1: Conformal prediction framework: split conformal prediction (top row) and conformalized quantile regression (bottom row)
• Figure 2: Schematic of selected microphysical processes (arrows) between the six hydrometeor categories (boxes) and water vapor in the two-moment microphysics scheme of seifertbehengTwomomentCloudMicrophysics2006. Horizontal (green) arrows represent interaction processes, vertical (blue) arrows represent phase transitions. For simplicity, riming is shown separately. If a process occurs more than once, arrows represent contributions to the total process rate. From an ML perspective, boxes represent input features (mass mixing ratios and number concentrations) and arrows represent targets (process rates)
• Figure 3: Calibrated prediction intervals with SCP and the NN (left) and CQR and the QNN (right) for autoconversion. For better visualization, we only show 1500 randomly selected samples
• Figure 4: Normalized mean prediction interval width (NMPIW) and prediction interval coverage probability (PICP) binned by value of the autoconversion rate for SCP (left) and CQR (right)
• Figure 5: Calibrated prediction intervals with split conformal prediction (SCP) for the accretion (NN), rain evaporation (RF), rain freezing (NN), melting to rain (NN) and total riming (NN) rate, obtained with the deterministic model that yields the best PICP (in brackets, see Table \ref{['table:results_picp']} in the main text). For better visualization, we only show 1500 randomly selected samples