Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PatchFormer: A Patch-Based Time Series Foundation Model with Hierarchical Masked Reconstruction and Cross-Domain Transfer Learning for Zero-Shot Multi-Horizon Forecasting

Olaf Yunus Laitinen Imanov, Derya Umut Kulali, Taner Yilmaz

TL;DR

PatchFormer proposes a patch-based time series foundation model that achieves strong zero-shot, multi-horizon forecasting across diverse domains by combining hierarchical patch tokenization, self-supervised masked reconstruction with contrastive learning, and cross-domain knowledge distillation. The model supports efficient deployment via adapter-based fine-tuning and demonstrates data efficiency, robust cross-domain transfer, and scalable pretraining benefits up to large data regimes. Empirically, PatchFormer delivers a substantial MSE reduction (27.3% on average) with far less task-specific data (around 94% reduction) and runs with 3.8× faster inference than full-sequence transformers, while maintaining robustness to missing data and excelling on long-horizon tasks. The work offers practical impact for cross-domain forecasting in climate, energy, traffic, finance, and healthcare, and opens avenues for multimodal and continual learning in time series foundation models.

Abstract

Time series forecasting is a fundamental problem with applications in climate, energy, healthcare, and finance. Many existing approaches require domain-specific feature engineering and substantial labeled data for each task. We introduce PatchFormer, a patch-based time series foundation model that uses hierarchical masked reconstruction for self-supervised pretraining and lightweight adapters for efficient transfer. PatchFormer segments time series into patches and learns multiscale temporal representations with learnable aggregation across temporal scales. Pretraining uses masked patch reconstruction with dynamic masking and objectives that encourage both local accuracy and global consistency, followed by cross-domain knowledge distillation. Experiments on 24 benchmark datasets spanning weather, energy, traffic, finance, and healthcare demonstrate state-of-the-art zero-shot multi-horizon forecasting, reducing mean squared error by 27.3 percent relative to strong baselines while requiring 94 percent less task-specific training data. The model exhibits near log-linear scaling with more pretraining data up to 100 billion points and processes length-512 sequences 3.8x faster than full-sequence transformers.

PatchFormer: A Patch-Based Time Series Foundation Model with Hierarchical Masked Reconstruction and Cross-Domain Transfer Learning for Zero-Shot Multi-Horizon Forecasting

TL;DR

PatchFormer proposes a patch-based time series foundation model that achieves strong zero-shot, multi-horizon forecasting across diverse domains by combining hierarchical patch tokenization, self-supervised masked reconstruction with contrastive learning, and cross-domain knowledge distillation. The model supports efficient deployment via adapter-based fine-tuning and demonstrates data efficiency, robust cross-domain transfer, and scalable pretraining benefits up to large data regimes. Empirically, PatchFormer delivers a substantial MSE reduction (27.3% on average) with far less task-specific data (around 94% reduction) and runs with 3.8× faster inference than full-sequence transformers, while maintaining robustness to missing data and excelling on long-horizon tasks. The work offers practical impact for cross-domain forecasting in climate, energy, traffic, finance, and healthcare, and opens avenues for multimodal and continual learning in time series foundation models.

Abstract

Time series forecasting is a fundamental problem with applications in climate, energy, healthcare, and finance. Many existing approaches require domain-specific feature engineering and substantial labeled data for each task. We introduce PatchFormer, a patch-based time series foundation model that uses hierarchical masked reconstruction for self-supervised pretraining and lightweight adapters for efficient transfer. PatchFormer segments time series into patches and learns multiscale temporal representations with learnable aggregation across temporal scales. Pretraining uses masked patch reconstruction with dynamic masking and objectives that encourage both local accuracy and global consistency, followed by cross-domain knowledge distillation. Experiments on 24 benchmark datasets spanning weather, energy, traffic, finance, and healthcare demonstrate state-of-the-art zero-shot multi-horizon forecasting, reducing mean squared error by 27.3 percent relative to strong baselines while requiring 94 percent less task-specific training data. The model exhibits near log-linear scaling with more pretraining data up to 100 billion points and processes length-512 sequences 3.8x faster than full-sequence transformers.
Paper Structure (28 sections, 15 equations, 2 figures, 7 tables)

This paper contains 28 sections, 15 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Classical and Deep Learning Forecasting
  4. Transformer Architectures for Time Series
  5. Patch-Based Sequence Modeling
  6. Time Series Foundation Models
  7. Methodology
  8. Problem Formulation
  9. Hierarchical Patch Tokenization
  10. Transformer Encoder
  11. Contrastive Masked Reconstruction
  12. Cross-Domain Knowledge Distillation
  13. Efficient Fine-Tuning with Adapters
  14. Experimental Setup
  15. Datasets
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Performance scaling with pretraining data size. MSE decreases log-linearly: $\text{MSE} = 0.412 - 0.045 \log_{10}(N)$ with $R^2 = 0.97$.
  • Figure 2: Few-shot learning on Weather dataset. PatchFormer (pretrained) reaches baseline performance with 500 samples (94% data reduction) vs. 10K for training from scratch.