Learning Robust Spectral Dynamics for Temporal Domain Generalization

TL;DR

FreKoo tackles Temporal Domain Generalization under complex concept drift by analyzing model parameter trajectories in the frequency domain. It decomposes trajectories into low-frequency components, modeled with a Koopman operator to extrapolate long-term trends and periodic patterns, and high-frequency components, which are regularized for smoothness to resist domain-specific noise. The method is supported by a multiscale generalization bound linking spectral dynamics and regularization to improved generalization. Empirically, FreKoo achieves state-of-the-art performance on six of seven TDG benchmarks, with notable strength on periodic drifts and real-world uncertainties, demonstrating robust cross-temporal generalization in dynamic environments.

Abstract

Modern machine learning models struggle to maintain performance in dynamic environments where temporal distribution shifts, \emph{i.e., concept drift}, are prevalent. Temporal Domain Generalization (TDG) seeks to enable model generalization across evolving domains, yet existing approaches typically assume smooth incremental changes, struggling with complex real-world drifts involving long-term structure (incremental evolution/periodicity) and local uncertainties. To overcome these limitations, we introduce FreKoo, which tackles these challenges via a novel frequency-domain analysis of parameter trajectories. It leverages the Fourier transform to disentangle parameter evolution into distinct spectral bands. Specifically, low-frequency component with dominant dynamics are learned and extrapolated using the Koopman operator, robustly capturing diverse drift patterns including both incremental and periodicity. Simultaneously, potentially disruptive high-frequency variations are smoothed via targeted temporal regularization, preventing overfitting to transient noise and domain uncertainties. In addition, this dual spectral strategy is rigorously grounded through theoretical analysis, providing stability guarantees for the Koopman prediction, a principled Bayesian justification for the high-frequency regularization, and culminating in a multiscale generalization bound connecting spectral dynamics to improved generalization. Extensive experiments demonstrate FreKoo's significant superiority over SOTA TDG approaches, particularly excelling in real-world streaming scenarios with complex drifts and uncertainties.

Figures (7)

• Figure 1: Illustration of challenges in TDG. Left: Complex drifting situations can involve both incremental shifts (e.g., $\mathcal{D}_{1} \rightarrow \mathcal{D}_{2} \rightarrow \mathcal{D}_{3}$) and long-term periodic returns (e.g., $\mathcal{D}_{t}$ resembling $\mathcal{D}_{1}$ after a cycle). The underlying optimal parameters $\theta_{t}$ evolve accordingly. Right: Within any domain $\mathcal{D}_{t}$, uncertainties or non-IID data concentrated in Areas 1, 2 and 4 compared to data in Area 3 can lead to local overfitting (solid blue line). Robust generalization requires converging to a smoother area (red dashed line) that is less sensitive to such localized noise or outliers.
• Figure 2: FreKoo Framework. It decomposes model parameter trajectories into low-frequency and high-frequency components via the Fourier transform. Then, Koopman operator is employed to learn the evolution of low-frequency dynamics, enabling robust prediction of incremental trend and periodicity. Also, it introduces a targeted temporal difference regularization to the high-frequency components, promoting smooth convergence and preventing overfitting to local uncertainties.
• Figure 3: Periodicity modeling performance on P-Moons dataset.
• Figure 4: Parameters sensitivity on 2-Moons and Appliance datasets.
• Figure 5: Visualization of domain-specific uncertainties on real-world datasets.

Theorems & Definitions (9)

• Theorem 1: Multiscale Generalization Bound
• Lemma 1: Koopman Stability
• Lemma 2: High-Frequency Smoothness Bias
• Lemma 3: Restatement of Lemma 1 - Koopman Stability
• Proof 1: Proof of Lemma \ref{['lemma:koopman_stability_restated']}
• Lemma 4: Restatement of Lemma 2 - High-Frequency Smoothness Bias
• Proof 2: Proof of Lemma \ref{['lemma:hf_smoothness_restated']}
• Theorem 2: Restatement of Theorem 1 - Multiscale Generalization Bound
• Proof 3: Proof of Theorem \ref{['theorem:generalization_bound_restated']}