Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture of Online and Offline Experts for Non-stationary Time Series

Zhilin Zhao, Longbing Cao, Yuanyu Wan

TL;DR

The paper tackles non-stationary time series with alternating offline intervals and a continuous online interval. It introduces Mixture of Online and Offline Experts (MOOE), which combines $K$ experts ( $K-1$ offline, 1 online) under a meta expert to adaptively weight predictions, enabling transfer of offline knowledge to the online window. The authors provide parameter convergence guarantees, dynamic regret bounds, and data-dependent generalization error bounds, establishing that MOOE achieves $O(1/\sqrt{T})$ generalization performance and can outperform purely online methods in dynamic environments. Empirically, MOOE delivers improved regret and predictive accuracy across synthetic and eight real-world non-stationary time-series datasets, including recurring drift and increasing noise scenarios, demonstrating practical impact in adapting to distribution shifts while controlling complexity.

Abstract

We consider a general and realistic scenario involving non-stationary time series, consisting of several offline intervals with different distributions within a fixed offline time horizon, and an online interval that continuously receives new samples. For non-stationary time series, the data distribution in the current online interval may have appeared in previous offline intervals. We theoretically explore the feasibility of applying knowledge from offline intervals to the current online interval. To this end, we propose the Mixture of Online and Offline Experts (MOOE). MOOE learns static offline experts from offline intervals and maintains a dynamic online expert for the current online interval. It then adaptively combines the offline and online experts using a meta expert to make predictions for the samples received in the online interval. Specifically, we focus on theoretical analysis, deriving parameter convergence, regret bounds, and generalization error bounds to prove the effectiveness of the algorithm.

Mixture of Online and Offline Experts for Non-stationary Time Series

TL;DR

The paper tackles non-stationary time series with alternating offline intervals and a continuous online interval. It introduces Mixture of Online and Offline Experts (MOOE), which combines experts ( offline, 1 online) under a meta expert to adaptively weight predictions, enabling transfer of offline knowledge to the online window. The authors provide parameter convergence guarantees, dynamic regret bounds, and data-dependent generalization error bounds, establishing that MOOE achieves generalization performance and can outperform purely online methods in dynamic environments. Empirically, MOOE delivers improved regret and predictive accuracy across synthetic and eight real-world non-stationary time-series datasets, including recurring drift and increasing noise scenarios, demonstrating practical impact in adapting to distribution shifts while controlling complexity.

Abstract

We consider a general and realistic scenario involving non-stationary time series, consisting of several offline intervals with different distributions within a fixed offline time horizon, and an online interval that continuously receives new samples. For non-stationary time series, the data distribution in the current online interval may have appeared in previous offline intervals. We theoretically explore the feasibility of applying knowledge from offline intervals to the current online interval. To this end, we propose the Mixture of Online and Offline Experts (MOOE). MOOE learns static offline experts from offline intervals and maintains a dynamic online expert for the current online interval. It then adaptively combines the offline and online experts using a meta expert to make predictions for the samples received in the online interval. Specifically, we focus on theoretical analysis, deriving parameter convergence, regret bounds, and generalization error bounds to prove the effectiveness of the algorithm.
Paper Structure (33 sections, 15 theorems, 81 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 33 sections, 15 theorems, 81 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning Theory for non-stationary Time Series
  4. Regret Analysis in dynamic environments
  5. Problem Statement
  6. Mixture of Online and Offline Experts
  7. Meta Expert
  8. Offline Expert
  9. Online Expert
  10. Theoretical Guarantees
  11. Parameter Convergence
  12. Regret Bound
  13. Generalization Error Bound
  14. Experimental Results
  15. Regret on Synthetic and Real-world Datasets
  16. ...and 18 more sections

Key Result

Lemma 1

$L_\mathcal{\widetilde{S}}^{\gamma}(\mathbf{w})$ is strongly-convex w.r.t. $\mathbf{w} \in \mathcal{W}$, and We set $\gamma \geq \left( \sum_{k = 1}^K \alpha_B^k L_{\mathcal{\widetilde{S}}}(\mathbf{w}_B^k) \right) / 4 R^2$ to ensure the validity of the regularization term.

Figures (3)

  • Figure 1: The working process of MOOE on non-stationary time series.
  • Figure 2: Regret and loss of MOOE and OGD methods.
  • Figure 3: MOOE with different $K_{\text{max}}$.

Theorems & Definitions (15)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Theorem 3
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • ...and 5 more