Predictive Inference in Multi-environment Scenarios

TL;DR

A novel resizing method to adapt to problem difficulty is demonstrated, which applies both to existing approaches for predictive inference and the methods developed; this reduces prediction set sizes using limited information from the test environment, a key to the methods' practical performance.

Abstract

We address the challenge of constructing valid confidence intervals and sets in problems of prediction across multiple environments. We investigate two types of coverage suitable for these problems, extending the jackknife and split-conformal methods to show how to obtain distribution-free coverage in such non-traditional, potentially hierarchical data-generating scenarios. We demonstrate a novel resizing method to adapt to problem difficulty, which applies both to existing approaches for predictive inference and the methods we develop; this reduces prediction set sizes using limited information from the test environment, a key to the methods' practical performance, which we evaluate through neurochemical sensing and species classification datasets. Our contributions also include extensions for settings with non-real-valued responses, a theory of consistency for predictive inference in these general problems, and insights on the limits of conditional coverage.

Figures (11)

• Figure 4.1: Influence of input $\delta$ on the performance of conformal algorithms applied to the neurochemical sensing data, with $\alpha = .05$. The plots show the empirical $1-\delta$, empirical $1-\alpha$, and empirical set length for both the split conformal and jackknife-minmax algorithms with various input $\delta$.
• Figure 4.2: Influence of input $\alpha$ on the performance of conformal algorithms the neurochemical sensing data. For these experiments, $\delta$ is set to be 0.33. The plots show the empirical $1-\delta$, empirical $1-\alpha$, and empirical set length for both the split conformal and jackknife-minmax algorithms with various input $\alpha$.
• Figure 4.3: Influence of input $\delta$ on the performance of conformal algorithms applied to the species classification data, with $\alpha = .05$. The plots show the empirical $1-\delta$, empirical $1-\alpha$, and empirical set length for both the split conformal and jackknife-minmax algorithms with various input $\delta$.
• Figure 4.4: Influence of input $\alpha$ on the performance of conformal algorithms the species classification data. For these experiments, $\delta$ is set to be 0.33. The plots show the empirical $1-\delta$, empirical $1-\alpha$, and empirical set length for both the split conformal and jackknife-minmax algorithms with various input $\alpha$.
• Figure 4.5: Performance of multi-environment jackknife-minmax and hierarchical jackknife+ applied to the neurochemical sensing data. Multi-environment jackknife-minmax takes in the parameters $\alpha, \delta$, and hierarchical jackknife+ takes in the parameter $\alpha$. For each value of $\alpha$, we find the largest $\delta$ such that the fraction of test samples covered by multi-environment split conformal exceeds that of hierarchical jackknife+.

Theorems & Definitions (33)

• definition 1
• definition 2
• definition 3: Quantile mappings
• theorem 1
• remark 1
• theorem 2
• corollary 1: Lee et al. LeeBaWi23, Theorems 1 and 5
• theorem 3
• theorem 4
• corollary 2