Scaling Up Temporal Domain Generalization via Temporal Experts Averaging

TL;DR

The paper addresses TDG under temporal distribution shifts by proposing Temporal Experts Averaging (TEA), a scalable weight-averaging framework that updates the entire model. TEA constructs a base model on all source domains, generates functionally diverse yet parameter-similar temporal experts via constrained finetuning, and forecasts a future-domain position in a low-dimensional weight space using PCA, followed by ARIMA-based trajectory prediction to compute adaptive averaging coefficients. The approach yields a TEA ensemble that minimizes future generalization error through a bias–variance–covariance–locality tradeoff, validated across 7 TDG benchmarks and 5 models, and extended to CDGTD with memory buffers. Experimental results show TEA achieving state-of-the-art performance, with large gains (up to 69%) and substantially lower training cost (up to 60x more efficient) than prior TDG methods, indicating strong practical potential for scalable temporal generalization. The work contributes theoretical BVCL insights, a concrete training-and-forecasting pipeline, and new evaluation benchmarks CLEAR-10/100 to push TDG research toward larger-scale, real-world scenarios.

Abstract

Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. Prior work often addresses this by predicting future model weights. However, full model prediction is prohibitively expensive for even reasonably sized models. Thus, recent methods only predict the classifier layer, limiting generalization by failing to adjust other model components. To address this, we propose Temporal Experts Averaging (TEA), a novel and scalable TDG framework that updates the entire model using weight averaging to maximize generalization potential while minimizing computational costs. Our theoretical analysis guides us to two steps that enhance generalization to future domains. First, we create expert models with functional diversity yet parameter similarity by fine-tuning a domain-agnostic base model on individual temporal domains while constraining weight changes. Second, we optimize the bias-variance tradeoff through adaptive averaging coefficients derived from modeling temporal weight trajectories in a principal component subspace. Expert's contributions are based on their projected proximity to future domains. Extensive experiments across 7 TDG benchmarks, 5 models, and 2 TDG settings shows TEA outperforms prior TDG methods by up to 69% while being up to 60x more efficient.

Figures (6)

• Figure 1: Examples of temporal domain generalization (TDG) span both (a) vision and (b) language tasks. TDG aims at enabling models trained on historical data to directly generalize to future data without retraining.
• Figure 2: TDG framework comparison. (a) Classifier-only TDGxie2024evolvingxie2024weight only predicts future classifiers to reduce computational costs in scaled-up scenarios, but limits generalization potential by neglecting other model components. (b) Our Temporal Expert Averaging (TEA) enables higher generalization potential by adjusting the entire model through predicting future averaging coefficients of temporal experts capturing diverse functionalities. The low-dimensional nature of these coefficients ensures TEA's efficiency in scaled-up scenarios.
• Figure 3: Overview of our TEA framework. Firstly, we obtain a base model $\theta_{\text{base}}$ through domain-agnostic pretraining on all source domains, then derive experts $\theta_1,...,\theta_n$ via constrained domain-specific incremental finetuning in reverse temporal order. Secondly, we apply PCA to expert weight deviations $\{\theta_i-\theta_{\text{base}}\}_{i=1}^{n}$, forecast future positions along the $P$ most significant components with Autoregressive Integrated Moving Average (ARIMA), effectively projecting experts into a low-dimensional space for prediction. Finally, we assign averaging coefficients based on projected expert-future proximity, where closer experts receive higher coefficients.
• Figure 4: Visualization of averaging coefficients and accuracies of experts on target domain $D_{S+1}$.
• Figure 5: An overview of our Only Adaptive Averaging ablation. (a) When optimizing the selector network in Only Adaptive Averaging, we use output averaging as a proxy task, utilizing the estimated coefficients to average the outputs of all snapshots. (b) During inference, we perform weight averaging with the optimized selector network.

Theorems & Definitions (8)

• Lemma 1: TWA and T-ENS
• proof
• Proposition 1: Bias-variance-covariance-locality decomposition for temporal weight averaging
• proof
• Lemma 2: Optimal Averaging Coefficients for Bias Minimization
• proof
• Lemma 3: Optimal Averaging Coefficients for Variance Minimization
• proof