Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks

Muhammad Anwar Ma'sum, MD Rasel Sarkar, Mahardhika Pratama, Savitha Ramasamy, Sreenatha Anavatti, Lin Liu, Habibullah, Ryszard Kowalczyk

TL;DR

MANTRA tackles dynamic long-term time-series forecasting by decoupling representation learning (slow learner) from rapid adaptation (an ensemble of fast learners) and fusing them through a universal representation transformer (URT). Built on a Autoformer-based backbone, it decomposes series into seasonal and trend components while URT enables quick concept drift adaptation with minimal parameter updates. Across four real datasets and multiple horizons, MANTRA yields consistent improvements over strong baselines in most settings (at least $3\%$ in many cases) and demonstrates robustness via ablations and sensitivity analyses. This approach offers a scalable, adaptable framework for nonstationary long-horizon forecasting with potential applicability to other backbone models.

Abstract

A reliable long-term time-series forecaster is highly demanded in practice but comes across many challenges such as low computational and memory footprints as well as robustness against dynamic learning environments. This paper proposes Meta-Transformer Networks (MANTRA) to deal with the dynamic long-term time-series forecasting tasks. MANTRA relies on the concept of fast and slow learners where a collection of fast learners learns different aspects of data distributions while adapting quickly to changes. A slow learner tailors suitable representations to fast learners. Fast adaptations to dynamic environments are achieved using the universal representation transformer layers producing task-adapted representations with a small number of parameters. Our experiments using four datasets with different prediction lengths demonstrate the advantage of our approach with at least $3\%$ improvements over the baseline algorithms for both multivariate and univariate settings. Source codes of MANTRA are publicly available in \url{https://github.com/anwarmaxsum/MANTRA}.

Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks

TL;DR

MANTRA tackles dynamic long-term time-series forecasting by decoupling representation learning (slow learner) from rapid adaptation (an ensemble of fast learners) and fusing them through a universal representation transformer (URT). Built on a Autoformer-based backbone, it decomposes series into seasonal and trend components while URT enables quick concept drift adaptation with minimal parameter updates. Across four real datasets and multiple horizons, MANTRA yields consistent improvements over strong baselines in most settings (at least in many cases) and demonstrates robustness via ablations and sensitivity analyses. This approach offers a scalable, adaptable framework for nonstationary long-horizon forecasting with potential applicability to other backbone models.

Abstract

A reliable long-term time-series forecaster is highly demanded in practice but comes across many challenges such as low computational and memory footprints as well as robustness against dynamic learning environments. This paper proposes Meta-Transformer Networks (MANTRA) to deal with the dynamic long-term time-series forecasting tasks. MANTRA relies on the concept of fast and slow learners where a collection of fast learners learns different aspects of data distributions while adapting quickly to changes. A slow learner tailors suitable representations to fast learners. Fast adaptations to dynamic environments are achieved using the universal representation transformer layers producing task-adapted representations with a small number of parameters. Our experiments using four datasets with different prediction lengths demonstrate the advantage of our approach with at least improvements over the baseline algorithms for both multivariate and univariate settings. Source codes of MANTRA are publicly available in \url{https://github.com/anwarmaxsum/MANTRA}.
Paper Structure (19 sections, 24 equations, 3 figures, 9 tables, 1 algorithm)

This paper contains 19 sections, 24 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Time-Series Forecasting
  4. Deep Learning for Dynamic Environments
  5. Preliminary
  6. Problem Formulation
  7. Autoformer
  8. Meta-Transformer Networks (MANTRA)
  9. Fast and Slow Learning Approach
  10. Drift Handling Mechanism via the URT Layer
  11. Complexity Analysis
  12. Experiments
  13. Datasets
  14. Implementation Details
  15. Baselines
  16. ...and 4 more sections

Figures (3)

  • Figure 1: MANTRA is consolidated by an ensemble of fast learners and a slow learner where the final output is aggregated by a URT layer. Fast learners and URT layer are trained in the end-to-end fashion minimizing the mean-squared error loss while slow learner is learned via the self-supervised learning loss, controlled reconstruction loss.
  • Figure 2: MANTRA and Autoformer validation loss on ETTm dataset
  • Figure 3: Visualization of MANTRA prediction compared to actual label on ETTm dataset