Conformal Prediction on Quantifying Uncertainty of Dynamic Systems

TL;DR

The paper tackles the challenge of quantifying uncertainty in neural operator models for dynamic PDE/video data. It adopts conformal prediction to generate prediction sets with finite-sample coverage guarantees, and compares CP against Monte Carlo dropout and ensemble methods on Navier–Stokes turbulence datasets, incorporating rotation-based symmetry tests. A calibration step using isotonic regression is employed to correct miscalibration in CP intervals, and performance is evaluated across multiple neural operators (e.g., FNO, TFNO, UNO) with metrics for sharpness, miscalibration, MAE, and RMSE. The results show that CP provides calibrated, theoretically grounded uncertainty estimates, outperforming baseline uncertainty methods in calibration while preserving predictive accuracy, and demonstrate that symmetry considerations can be integrated into the calibration process. This work advances trustworthy uncertainty quantification for time-evolving physical systems, with implications for video prediction, weather forecasting, and scientific computing.

Abstract

Numerous studies have focused on learning and understanding the dynamics of physical systems from video data, such as spatial intelligence. Artificial intelligence requires quantitative assessments of the uncertainty of the model to ensure reliability. However, there is still a relative lack of systematic assessment of the uncertainties, particularly the uncertainties of the physical data. Our motivation is to introduce conformal prediction into the uncertainty assessment of dynamical systems, providing a method supported by theoretical guarantees. This paper uses the conformal prediction method to assess uncertainties with benchmark operator learning methods. We have also compared the Monte Carlo Dropout and Ensemble methods in the partial differential equations dataset, effectively evaluating uncertainty through straight roll-outs, making it ideal for time-series tasks.

Figures (10)

• Figure 1: The model is evaluated using three methods: (a) the ensemble method, (b) conformal prediction, and (c) Monte Carlo dropout.
• Figure 2: Results of CP with $\alpha=0.05$ by FNO$_{128}$. Specifically, the first row corresponds to the ground truth, the second row is the model outputs, the third row is the absolute difference, while the fourth row, denoted as $\frac{Q_{1-\alpha}}{1.96}$, depicts the standard deviation across time.
• Figure 3: The calibration results of CP. From left to right, the subfigure respectively represents the ordered prediction interval, the average calibration curve, and the re-calibrated curve after isotonic regression..
• Figure 4: Results of MC dropout by FNO$_{128}$. Specifically, the first row corresponds to the ground truth, the second row is the model outputs, the third row is the absolute difference, and the fourth row depicts the standard deviation across time.
• Figure 5: The calibration results of MC dropout. From left to right, the subfigure respectively represents the ordered prediction interval, the average calibration curve, and the re-calibrated curve after isotonic regression.