Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VE: Modeling Multivariate Time Series Correlation with Variate Embedding

Shangjiong Wang, Zhihong Man, Zhenwei Cao, Jinchuan Zheng, Zhikang Ge

TL;DR

The variate embedding (VE) pipeline is presented, which learns a unique and consistent embedding for each variate and combines it with Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) techniques to enhance forecasting performance while controlling parameter size.

Abstract

Multivariate time series forecasting relies on accurately capturing the correlations among variates. Current channel-independent (CI) models and models with a CI final projection layer are unable to capture these dependencies. In this paper, we present the variate embedding (VE) pipeline, which learns a unique and consistent embedding for each variate and combines it with Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) techniques to enhance forecasting performance while controlling parameter size. The VE pipeline can be integrated into any model with a CI final projection layer to improve multivariate forecasting. The learned VE effectively groups variates with similar temporal patterns and separates those with low correlations. The effectiveness of the VE pipeline is demonstrated through experiments on four widely-used datasets. The code is available at: https://github.com/swang-song/VE.

VE: Modeling Multivariate Time Series Correlation with Variate Embedding

TL;DR

The variate embedding (VE) pipeline is presented, which learns a unique and consistent embedding for each variate and combines it with Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) techniques to enhance forecasting performance while controlling parameter size.

Abstract

Multivariate time series forecasting relies on accurately capturing the correlations among variates. Current channel-independent (CI) models and models with a CI final projection layer are unable to capture these dependencies. In this paper, we present the variate embedding (VE) pipeline, which learns a unique and consistent embedding for each variate and combines it with Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) techniques to enhance forecasting performance while controlling parameter size. The VE pipeline can be integrated into any model with a CI final projection layer to improve multivariate forecasting. The learned VE effectively groups variates with similar temporal patterns and separates those with low correlations. The effectiveness of the VE pipeline is demonstrated through experiments on four widely-used datasets. The code is available at: https://github.com/swang-song/VE.
Paper Structure (11 sections, 4 equations, 3 figures, 3 tables)

This paper contains 11 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. VE Pipeline
  4. VEMoE
  5. LoRA
  6. Experiments
  7. Experimental Setup
  8. Main Results
  9. Analysis of learned variate embedding
  10. Abalation study
  11. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed method. Our VE pipeline can be integrated into any model where the final projection layer is channel-independent. In our VE pipeline, we generate an embedding of dimension $k$ for each variate. Each embedding is then processed by a $softmax$ function, and the results are used as weights for a set of linear weights, which are decomposed into two low-rank matrices to reduce the parameter size. The weighted linear weights are then used to do projection for each variate, replacing the original channel-independent weights.
  • Figure 2: Visualization of VE similarity for FITS($H=96$) on mixed dataset. Left: absolute cosine similarity of the learned VE; middle: a zoomed-in view of the similaryty matrix for variates from 351 to 355; right: the data points of the five variates in the first sample of the first batch in the testing dataset.
  • Figure 3: Visualization of the magnitude of the weighted complex-value wegiths of FITS($H=96$) for variates from 353 to 355.