Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift
Mouxiang Chen, Lefei Shen, Han Fu, Zhuo Li, Jianling Sun, Chenghao Liu
TL;DR
This work identifies context-driven distribution shift (CDS) as a core challenge in time-series forecasting, where observed and unobserved temporal contexts bias predictions. It introduces Reconditionor, a residual-based detector that uses mutual information between prediction residuals and contexts, and SOLID, a sample-level contextualized adapter that tunes only the prediction layer on a contextually similar dataset to mitigate CDS. Theoretical analysis shows a bias-variance trade-off guiding the calibration, and extensive experiments across eight real-world datasets demonstrate that Reconditionor reliably signals CDS and SOLID consistently improves forecasting performance, especially under strong CDS, with modest overhead. The framework is model-agnostic and can be layered onto various TSF architectures, offering a practical path to robust predictions in non-stationary, context-dependent environments.
Abstract
Recent years have witnessed the success of introducing deep learning models to time series forecasting. From a data generation perspective, we illustrate that existing models are susceptible to distribution shifts driven by temporal contexts, whether observed or unobserved. Such context-driven distribution shift (CDS) introduces biases in predictions within specific contexts and poses challenges for conventional training paradigms. In this paper, we introduce a universal calibration methodology for the detection and adaptation of CDS with a trained model. To this end, we propose a novel CDS detector, termed the "residual-based CDS detector" or "Reconditionor", which quantifies the model's vulnerability to CDS by evaluating the mutual information between prediction residuals and their corresponding contexts. A high Reconditionor score indicates a severe susceptibility, thereby necessitating model adaptation. In this circumstance, we put forth a straightforward yet potent adapter framework for model calibration, termed the "sample-level contextualized adapter" or "SOLID". This framework involves the curation of a contextually similar dataset to the provided test sample and the subsequent fine-tuning of the model's prediction layer with a limited number of steps. Our theoretical analysis demonstrates that this adaptation strategy can achieve an optimal bias-variance trade-off. Notably, our proposed Reconditionor and SOLID are model-agnostic and readily adaptable to a wide range of models. Extensive experiments show that SOLID consistently enhances the performance of current forecasting models on real-world datasets, especially on cases with substantial CDS detected by the proposed Reconditionor, thus validating the effectiveness of the calibration approach.