Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Conformal Prediction Regions with Simple Structures by Optimizing Convex Shape Templates

Renukanandan Tumu, Matthew Cleaveland, Rahul Mangharam, George J. Pappas, Lars Lindemann

TL;DR

This work addresses the conservatism and complexity of conformal prediction regions by learning parameterized convex shape templates to produce multi-modal, convex CP regions with minimal volume while maintaining the coverage guarantee $1-\delta$. It clusters residuals using KDE and Mean Shift, fits separate shape templates per cluster (ellipsoid, convex hull, hyper-rectangle) by volume-minimization, and defines a normalized non-conformity score $R_\theta$ to enable standard CP on a held-out calibration set. A time-series extension combines per-time residuals into a single CP score, enabling valid multi-step prediction regions. Experiments on F16 fighter-jet trajectories and autonomous-vehicle trajectories show up to about $68\%$ reduction in prediction-region area compared with L2-based baselines, and the authors provide an open-source toolbox for practical deployment.

Abstract

Conformal prediction is a statistical tool for producing prediction regions for machine learning models that are valid with high probability. A key component of conformal prediction algorithms is a \emph{non-conformity score function} that quantifies how different a model's prediction is from the unknown ground truth value. Essentially, these functions determine the shape and the size of the conformal prediction regions. While prior work has gone into creating score functions that produce multi-model prediction regions, such regions are generally too complex for use in downstream planning and control problems. We propose a method that optimizes parameterized \emph{shape template functions} over calibration data, which results in non-conformity score functions that produce prediction regions with minimum volume. Our approach results in prediction regions that are \emph{multi-modal}, so they can properly capture residuals of distributions that have multiple modes, and \emph{practical}, so each region is convex and can be easily incorporated into downstream tasks, such as a motion planner using conformal prediction regions. Our method applies to general supervised learning tasks, while we illustrate its use in time-series prediction. We provide a toolbox and present illustrative case studies of F16 fighter jets and autonomous vehicles, showing an up to $68\%$ reduction in prediction region area compared to a circular baseline region.

Multi-Modal Conformal Prediction Regions with Simple Structures by Optimizing Convex Shape Templates

TL;DR

This work addresses the conservatism and complexity of conformal prediction regions by learning parameterized convex shape templates to produce multi-modal, convex CP regions with minimal volume while maintaining the coverage guarantee . It clusters residuals using KDE and Mean Shift, fits separate shape templates per cluster (ellipsoid, convex hull, hyper-rectangle) by volume-minimization, and defines a normalized non-conformity score to enable standard CP on a held-out calibration set. A time-series extension combines per-time residuals into a single CP score, enabling valid multi-step prediction regions. Experiments on F16 fighter-jet trajectories and autonomous-vehicle trajectories show up to about reduction in prediction-region area compared with L2-based baselines, and the authors provide an open-source toolbox for practical deployment.

Abstract

Conformal prediction is a statistical tool for producing prediction regions for machine learning models that are valid with high probability. A key component of conformal prediction algorithms is a \emph{non-conformity score function} that quantifies how different a model's prediction is from the unknown ground truth value. Essentially, these functions determine the shape and the size of the conformal prediction regions. While prior work has gone into creating score functions that produce multi-model prediction regions, such regions are generally too complex for use in downstream planning and control problems. We propose a method that optimizes parameterized \emph{shape template functions} over calibration data, which results in non-conformity score functions that produce prediction regions with minimum volume. Our approach results in prediction regions that are \emph{multi-modal}, so they can properly capture residuals of distributions that have multiple modes, and \emph{practical}, so each region is convex and can be easily incorporated into downstream tasks, such as a motion planner using conformal prediction regions. Our method applies to general supervised learning tasks, while we illustrate its use in time-series prediction. We provide a toolbox and present illustrative case studies of F16 fighter jets and autonomous vehicles, showing an up to reduction in prediction region area compared to a circular baseline region.
Paper Structure (12 sections, 2 theorems, 16 equations, 3 figures)

This paper contains 12 sections, 2 theorems, 16 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries: Conformal Prediction Regions
  3. Computing Convex Multi-Modal Conformal Prediction Regions
  4. Density Estimation
  5. Clustering
  6. Shape Construction
  7. Conformalization
  8. Dealing with Time-Series Data
  9. Simulations
  10. F16
  11. Vehicle Trajectory Prediction
  12. Conclusion

Key Result

theorem 1

Let the conditions from Problem prob hold. Let $R_{S_c}$ be the non-conformity score function according to equation eq:nonconformFuncShapes where the parameters $\theta_1,\hdots,\theta_K$ are obtained by solving eq:general_shape_optim. Define $R:=R_{S_c}(Y-\hat{Y})$ for the random variable $(X,Y)\si where $C\coloneqq Quantile(\{ R_{n_1+1},\hdots,R_{n},\infty \}, 1-\delta)$.

Figures (3)

  • Figure 1: Vehicle Trajectory Prediction regions, $S_{CP}$, plotted alongside benchmark prediction regions $S_{\text{benchmark}}$, which are based on the 2-norm of the residual. All methods achieve the target $90\%$ coverage rate. The Convex Hull, Hyperrectangle, and Ellipsoid Regions are $68.92\%$, $59.43\%$, and $66.92\%$ smaller respectively.
  • Figure 2: (a) Shows the coverage rates over 1,000 random splits of $D_{cal,2}$ and $D_{val}$ for the F16 example. (b) Shows fit conformal regions for the F16 example. (c) Shows the coverage rates over 10,000 random splits of $D_{cal,2}$ and $D_{val}$ for the car example. (d) Shows an example of the prediction regions shown on an actual prediction of the trajectories.
  • Figure 3: This figure shows the size of conformal prediction regions created for a time series prediction of vehicle motion over fifty timesteps. We generate a prediction region for only the timesteps shown, using the method in cleaveland2023conformal, labeled the LCP region. We also generate conformal prediction regions using our method, which are shown in black. Each figure includes the area of the regions shown, and all methods achieve the desired coverage.

Theorems & Definitions (2)

  • theorem 1
  • corollary 1