General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data

TL;DR

This work addresses universal knowledge representation for multivariate time series by introducing GTM, a decoder-only, frequency-aware architecture that explicitly captures time granularity in both temporal and frequency domains. A Fourier Knowledge Attention mechanism enables time granularity-aware representations, while an autoregressive blank infilling pre-training scheme unifies learning across forecasting, imputation, and anomaly detection. GTM is pre-trained on UTSD-12G and evaluated across diverse downstream tasks, outperforming SOTA baselines and showing consistent gains over non-pretrained variants. The results underscore the value of integrating time granularity with frequency-domain learning to improve adaptability and performance in multi-task MTS analysis, with practical implications for scalable, task-agnostic time-series models.

Abstract

Universal knowledge representation is a central problem for multivariate time series(MTS) foundation models and yet remains open. This paper investigates this problem from the first principle and it makes four folds of contributions. First, a new empirical finding is revealed: time series with different time granularities (or corresponding frequency resolutions) exhibit distinct joint distributions in the frequency domain. This implies a crucial aspect of learning universal knowledge, one that has been overlooked by previous studies. Second, a novel Fourier knowledge attention mechanism is proposed to enable learning time granularity-aware representations from both the temporal and frequency domains. Third, an autoregressive blank infilling pre-training framework is incorporated to time series analysis for the first time, leading to a generative tasks agnostic pre-training strategy. To this end, we develop the General Time-series Model (GTM), a unified MTS foundation model that addresses the limitation of contemporary time series models, which often require token, pre-training, or model-level customizations for downstream tasks adaption. Fourth, extensive experiments show that GTM outperforms state-of-the-art (SOTA) methods across all generative tasks, including long-term forecasting, anomaly detection, and imputation.

Figures (6)

• Figure 1: 3D visualization of the Amplitude-Freq. joint dist. in the freq. domain for MTS data with varying time granularities.
• Figure 2: GTM model architecture for pre-training. Left: MTS data pass through three key components—input embedding, N-stack Transformer backbone, and output projection—to generate reconstruction results autoregressively. Lower right: Patching and masking processes using both full attention and causal attention mechanisms, adapted from the NLP field and optimized for MTS pre-training. Upper right: A novel knowledge attention module designed to learn representations of MTS data with varying time granularities.
• Figure 3: Phae-Frequency distribution of time series data with various granularities.
• Figure 4: Visualization of forecasting results.
• Figure 5: Visualization of imputation results.