VarDrop: Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting

TL;DR

This work addresses the quadratic cost of attention in variate-tokenized Transformers for multivariate time series by introducing VarDrop, which adaptively drops redundant variate tokens during training. It achieves this via k-dominant frequency hashing ($k$-DFH) to group variates by similar periodic content, followed by stratified sampling and sparse attention on a reduced token set; the resulting complexity scales as $O\left(N^2 (1-\delta)^2 d\right)$, with $\delta$ the token-reduction rate. Experiments on Electricity, Traffic, Weather, and Solar-Energy show strong performance gains over efficient baselines, with significant token reductions (up to ~85%) and reduced training time and memory footprints, while maintaining close to ground-truth accuracy. The method is modular and architecture-agnostic, enabling easy integration with existing variate-tokenized Transformers to enable scalable forecasting on high-dimensional data.

Abstract

Variate tokenization, which independently embeds each variate as separate tokens, has achieved remarkable improvements in multivariate time series forecasting. However, employing self-attention with variate tokens incurs a quadratic computational cost with respect to the number of variates, thus limiting its training efficiency for large-scale applications. To address this issue, we propose VarDrop, a simple yet efficient strategy that reduces the token usage by omitting redundant variate tokens during training. VarDrop adaptively excludes redundant tokens within a given batch, thereby reducing the number of tokens used for dot-product attention while preserving essential information. Specifically, we introduce k-dominant frequency hashing (k-DFH), which utilizes the ranked dominant frequencies in the frequency domain as a hash value to efficiently group variate tokens exhibiting similar periodic behaviors. Then, only representative tokens in each group are sampled through stratified sampling. By performing sparse attention with these selected tokens, the computational cost of scaled dot-product attention is significantly alleviated. Experiments conducted on public benchmark datasets demonstrate that VarDrop outperforms existing efficient baselines.

Figures (10)

• Figure 1: Variate redundancy in public periodic time series datasets. Most variates are highly correlated with others. For more detailed analysis, please refer to Appendix A.
• Figure 2: Overall procedure of VarDrop. Given a batch of multivariate time series, the hash values representing the top-$k$ amplitudes for each variate are generated through $k$-Dominant Frequency Hashing ($k$-DFH). Then, stratified sampling is conducted based on groups of variates that share the same hash value to omit redundant variates. Finally, sparse attention is performed on the reduced set of variate tokens, enabling efficient training.
• Figure 3: Correlation matrix and corresponding sparse matrices with varying $gs$ values on the Electricity dataset.
• Figure 4: Forecasting results with input-96-predict-96 setting with their token reduction ratios for varying two hyperparameters $k \in \{2,3,4,5\}$ and $gs \in \{1, \ldots, 15\}$ on the Electricity dataset.
• Figure 5: Visual examples of variates grouped by $k$-DFH. The red lines represent the mean values of the variates within each group, illustrating the distinctive periodic patterns. The hash values represent the ordered set of $k$ dominant frequencies.

Theorems & Definitions (3)

• Definition 3.1: $k$-Dominant Frequency
• Theorem 3.2: Error of $k$-DFH Approximation
• proof