HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts

TL;DR

<3-5 sentence high-level summary> HyperTime addresses temporal distribution shifts by optimizing hyperparameters under a lexicographic objective that jointly considers average validation performance and worst-case validation performance across chronologically partitioned validation sets. The method is model-agnostic and complements robust training, with theoretical bounds on test loss and strong empirical results on gradient-boosting trees and neural networks across multiple temporally shifted datasets. It demonstrates that a time-aware, worst-case-centric HPO strategy can yield more temporally robust models than traditional single-objective HPO or standard training. The work also provides practical validation-set construction guidance and shows compatibility with existing robust-learning techniques to further boost performance.</3-5 sentence high-level summary>

Abstract

In this work, we propose a hyperparameter optimization method named \emph{HyperTime} to find hyperparameters robust to potential temporal distribution shifts in the unseen test data. Our work is motivated by an important observation that it is, in many cases, possible to achieve temporally robust predictive performance via hyperparameter optimization. Based on this observation, we leverage the `worst-case-oriented' philosophy from the robust optimization literature to help find such robust hyperparameter configurations. HyperTime imposes a lexicographic priority order on average validation loss and worst-case validation loss over chronological validation sets. We perform a theoretical analysis on the upper bound of the expected test loss, which reveals the unique advantages of our approach. We also demonstrate the strong empirical performance of the proposed method on multiple machine learning tasks with temporal distribution shifts.

Figures (9)

• Figure 1: Validation loss vs. test loss on the Electricity dataset, where the validation and test data are from different time periods. Each point is a hyperparameter configuration randomly sampled from the search space. The loss here is (1- ROC_AUC).
• Figure 2: Chronological validation data sets construction with Cross Validation and Holdout strategies.
• Figure 3: Per fold test loss (lower the better) for tuning gradient-boosting trees on different datasets. The results are averaged over different random seeds. The results are from the same set of experiments with that in Table \ref{['tab:tabular']}.
• Figure 4: Test loss of CFO and HyperTime on different folds with/without using chronological validation sets.
• Figure 5: Test loss of different folds using HyperTime, HyperTime_Reverse, CFO_WeightedCombine and CFO_Worst.

Theorems & Definitions (5)

• Theorem 1
• Remark 4.1: The role of $\kappa$
• Lemma 2
• proof : Proof of Lemma \ref{['lemma:difference_between_empirical_and_expected']}
• proof : Proof of Theorem \ref{['theorem_bound']}