Online Conformal Model Selection for Nonstationary Time Series

TL;DR

The paper tackles online model selection for nonstationary time series by introducing the Model Prediction Set (MPS), a framework that maintains a confidence set $\mathcal{C}_t$ of candidate models to cover the next-period best model $\mathcal{M}_{t+1}$ with long-run miscoverage $\bar{\alpha}$. MPS blends Model Confidence Set (MCS) assessments with Bellman conformal inference (BCI) style calibration to adapt the instantaneous miscoverage $\alpha_t$, yielding nonasymptotic guarantees and robustness to unknown nonstationary forms. Through synthetic simulations and empirical studies on OT and VIX data, MPS demonstrates reliable miscoverage control near $1-\bar{\alpha}$, while frequently producing small, informative “quality sets” that reveal evolving dynamics and model competition; offline MCS, by contrast, often yields full sets or poor adaptability. The framework is highly general, applicable to any data-generating process, data structure, model class, training method, and evaluation metric, offering practical, real-time insights for nonstationary environments and beyond forecasting tasks.

Abstract

This paper introduces the MPS (Model Prediction Set), a novel framework for online model selection for nonstationary time series. Classical model selection methods, such as information criteria and cross-validation, rely heavily on the stationarity assumption and often fail in dynamic environments which undergo gradual or abrupt changes over time. Yet real-world data are rarely stationary, and model selection under nonstationarity remains a largely open problem. To tackle this challenge, we combine conformal inference with model confidence sets to develop a procedure that adaptively selects models best suited to the evolving dynamics at any given time. Concretely, the MPS updates in real time a confidence set of candidate models that covers the best model for the next time period with a specified long-run probability, while adapting to nonstationarity of unknown forms. Through simulations and real-world data analysis, we demonstrate that MPS reliably and efficiently identifies optimal models under nonstationarity, an essential capability lacking in offline methods. Moreover, MPS frequently produces high-quality sets with small cardinality, whose evolution offers deeper insights into changing dynamics. As a generic framework, MPS accommodates any data-generating process, data structure, model class, training method, and evaluation metric, making it broadly applicable across diverse problem settings.

Figures (5)

• Figure 1: Miscoverage rates (i.e., the proportion of times the best model $\mathcal{M}_{t+1}$ is not included in $\mathcal{C}_t$ up to time $t$ evaluated over a moving window of size 100) of several model selection methods applied to forecasting with the ETTh1 dataset (see Section \ref{['sec:data']}). We compare: (i) offline single-model selection approaches (AIC, BIC, and CV based on minimizing forecast error over a hold-out set); (ii) offline MCS; and (iii) the proposed MPS. The best model is defined using one-step-ahead forecast error (forecasting time $t+1$ based on data up to time $t$) as the evaluation metric. As shown, only MPS achieves accurate control of miscoverage close to the nominal level $0.2$. All single-model selection methods perform poorly, and MCS exhibits extremely low miscoverage by producing trivial sets that ultimately include all candidate models. See Appendix \ref{['sec:a1']} for experiment details and Section \ref{['sec:data']} for a more detailed comparison of MPS and MCS.
• Figure 2: Illustration of the MPS procedure. More details are provided in Section \ref{['sec:method']}.
• Figure 3: Miscoverage rate, cardinality, and the range of losses (i.e., the values of the model evaluation metric) for the models selected by MPS and MCS for simulation experiments. Results are shown for three loss matrix designs (a)--(c), as well as for the model fitting experiment in panel (d).
• Figure 4: Comparison of MPS and MCS performance on real-world data: (a) OT and (b) VIX.
• Figure 5: Time series plots of (a) OT and (b) VIX. The dashed lines indicate the end of the initial training set at $n=240$.

Theorems & Definitions (3)

• Theorem 1
• proof
• Remark 1