Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online time series prediction using feature adjustment

Xiannan Huang, Shuhan Qiu, Jiayuan Du, Chao Yang

TL;DR

ADAPT-Z (Automatic Delta Adjustment via Persistent Tracking in Z-space) is introduced, proposing that distribution shifts stem from changes in underlying latent factors influencing the data and updating the feature representations of these latent factors may be more effective.

Abstract

Time series forecasting is of significant importance across various domains. However, it faces significant challenges due to distribution shift. This issue becomes particularly pronounced in online deployment scenarios where data arrives sequentially, requiring models to adapt continually to evolving patterns. Current time series online learning methods focus on two main aspects: selecting suitable parameters to update (e.g., final layer weights or adapter modules) and devising suitable update strategies (e.g., using recent batches, replay buffers, or averaged gradients). We challenge the conventional parameter selection approach, proposing that distribution shifts stem from changes in underlying latent factors influencing the data. Consequently, updating the feature representations of these latent factors may be more effective. To address the critical problem of delayed feedback in multi-step forecasting (where true values arrive much later than predictions), we introduce ADAPT-Z (Automatic Delta Adjustment via Persistent Tracking in Z-space). ADAPT-Z utilizes an adapter module that leverages current feature representations combined with historical gradient information to enable robust parameter updates despite the delay. Extensive experiments demonstrate that our method consistently outperforms standard base models without adaptation and surpasses state-of-the-art online learning approaches across multiple datasets.

Online time series prediction using feature adjustment

TL;DR

ADAPT-Z (Automatic Delta Adjustment via Persistent Tracking in Z-space) is introduced, proposing that distribution shifts stem from changes in underlying latent factors influencing the data and updating the feature representations of these latent factors may be more effective.

Abstract

Time series forecasting is of significant importance across various domains. However, it faces significant challenges due to distribution shift. This issue becomes particularly pronounced in online deployment scenarios where data arrives sequentially, requiring models to adapt continually to evolving patterns. Current time series online learning methods focus on two main aspects: selecting suitable parameters to update (e.g., final layer weights or adapter modules) and devising suitable update strategies (e.g., using recent batches, replay buffers, or averaged gradients). We challenge the conventional parameter selection approach, proposing that distribution shifts stem from changes in underlying latent factors influencing the data. Consequently, updating the feature representations of these latent factors may be more effective. To address the critical problem of delayed feedback in multi-step forecasting (where true values arrive much later than predictions), we introduce ADAPT-Z (Automatic Delta Adjustment via Persistent Tracking in Z-space). ADAPT-Z utilizes an adapter module that leverages current feature representations combined with historical gradient information to enable robust parameter updates despite the delay. Extensive experiments demonstrate that our method consistently outperforms standard base models without adaptation and surpasses state-of-the-art online learning approaches across multiple datasets.

Paper Structure

This paper contains 34 sections, 1 theorem, 15 equations, 14 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Key Idea
  5. Learning adjust term using adapter
  6. Computing Historical Gradients for Adapter Input
  7. Online update
  8. Experiments
  9. Set up
  10. Baseline methods
  11. Results
  12. Main Results
  13. Finetune using training set
  14. Analysis of feature location
  15. Ablation Experiments
  16. ...and 19 more sections

Key Result

Theorem C.1

Under the following assumptions: the dynamic regret is bounded by:

Figures (14)

  • Figure 1: The difference between Our method (ADAPT-Z) and other methods
  • Figure 2: The structure of iTransformer
  • Figure 3: The relationship between the number of training epochs on the validation set and the final MSE during deployment. The yellow star marks the epoch number that achieved the optimal MSE. The black dashed line shows the MSE when using the original model directly on the test set. The red dashed line represents the MSE of the best baseline online deployment method. The MSE is the average result of three prediction horizons.
  • Figure 4: The structure of adapter
  • Figure 5: The relationship between batch size and final MSE across three datasets under varying prediction horizons. Each row of subplots corresponds to one dataset at different prediction horizons. And each column represents results for a fixed prediction horizon across all three datasets.
  • ...and 9 more figures

Theorems & Definitions (2)

  • Theorem C.1
  • Proof C.1