Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting

Hao Chen, Tao Han, Jie Zhang, Song Guo, Fenghua Ling, Lei Bai

TL;DR

The paper tackles long-horizon weather forecasting by addressing error accumulation and training cost through a novel three-stage pipeline. The core methods are EMFormer with a fused Multi-Convs Layer, accumulative context finetuning with memory pruning, and a sinusoidal loss balancing latitude-weighted and variable-weighted objectives. Empirical results show improved long-term forecast accuracy, strong typhoon-track prediction, and competitive performance on ImageNet-1K and ADE20K, with substantial efficiency gains over traditional multi-scale modules. The approach generalizes beyond meteorology to vision benchmarks, indicating broad applicability of efficient multi-scale transformers with memory-augmented long-context modeling.

Abstract

Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend prediction horizons, they remain constrained by the issues of catastrophic forgetting, error accumulation, and high training overhead. To address these limitations, we present a novel pipeline across pretraining, finetuning and forecasting to enhance long-context modeling while reducing computational overhead. First, we introduce an Efficient Multi-scale Transformer (EMFormer) to extract multi-scale features through a single convolution in both training and inference. Based on the new architecture, we further employ an accumulative context finetuning to improve temporal consistency without degrading short-term accuracy. Additionally, we propose a composite loss that dynamically balances different terms via a sinusoidal weighting, thereby adaptively guiding the optimization trajectory throughout pretraining and finetuning. Experiments show that our approach achieves strong performance in weather forecasting and extreme event prediction, substantially improving long-term forecast accuracy. Moreover, EMFormer demonstrates strong generalization on vision benchmarks (ImageNet-1K and ADE20K) while delivering a 5.69x speedup over conventional multi-scale modules.

EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting

TL;DR

The paper tackles long-horizon weather forecasting by addressing error accumulation and training cost through a novel three-stage pipeline. The core methods are EMFormer with a fused Multi-Convs Layer, accumulative context finetuning with memory pruning, and a sinusoidal loss balancing latitude-weighted and variable-weighted objectives. Empirical results show improved long-term forecast accuracy, strong typhoon-track prediction, and competitive performance on ImageNet-1K and ADE20K, with substantial efficiency gains over traditional multi-scale modules. The approach generalizes beyond meteorology to vision benchmarks, indicating broad applicability of efficient multi-scale transformers with memory-augmented long-context modeling.

Abstract

Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend prediction horizons, they remain constrained by the issues of catastrophic forgetting, error accumulation, and high training overhead. To address these limitations, we present a novel pipeline across pretraining, finetuning and forecasting to enhance long-context modeling while reducing computational overhead. First, we introduce an Efficient Multi-scale Transformer (EMFormer) to extract multi-scale features through a single convolution in both training and inference. Based on the new architecture, we further employ an accumulative context finetuning to improve temporal consistency without degrading short-term accuracy. Additionally, we propose a composite loss that dynamically balances different terms via a sinusoidal weighting, thereby adaptively guiding the optimization trajectory throughout pretraining and finetuning. Experiments show that our approach achieves strong performance in weather forecasting and extreme event prediction, substantially improving long-term forecast accuracy. Moreover, EMFormer demonstrates strong generalization on vision benchmarks (ImageNet-1K and ADE20K) while delivering a 5.69x speedup over conventional multi-scale modules.
Paper Structure (47 sections, 4 theorems, 40 equations, 20 figures, 20 tables, 3 algorithms)

This paper contains 47 sections, 4 theorems, 40 equations, 20 figures, 20 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. EMFormer
  5. Accumulative Context Finetuning
  6. Loss Function
  7. Experiments
  8. Dataset and Implementation Details
  9. Main Results
  10. Ablation Study
  11. Conclusion
  12. Proof of Theorem
  13. : Equivalence and Efficiency of the Multi-Conv Layer
  14. Preliminaries and Notation
  15. Proof of 1: Functional Equivalence (Forward Pass)
  16. ...and 32 more sections

Key Result

Theorem 2.1

Let $\mathcal{M}_{\text{plain}}$ denote a standard multi-scale module with kernels $K_{r \in \{1,3,5\}}$, and let $\mathcal{M}_{\text{mc}}$ denote the multi-convs layer. Given identical input features $\mathbf{Z}^t$ and identical kernel initialization, the following properties hold: 1. Function equi 2. Gradient equivalence: The gradients to each weight $K_r$ are identical: 3. Computational effi

Figures (20)

  • Figure 1: Denormalized Z500 RMSE ($m^2/s^2$) for short-term (6-hour) and medium-term (5-day) forecasts. (a) Training convergence comparison in 6-step finetuning: accumulative context finetuning and previous finetuning. (b) Medium-term forecast performance: The proposed method consistently outperforms VA-MoE across pretraining and multi-step finetuning (6, 8, 10 steps). Two models are trained by us with A100.
  • Figure 2: Illustration of the novel pipeline with three stages. Stage 1: EMFormer is pretrained on atmospheric variables with pruning-recovering architecture that includes a pruning module, a series of EMFormer blocks, and a recovering module; Stage 2: accumulative context finetuning; Stage 3: The forecasting stage with weather forecasting and typhoon track prediction.
  • Figure 3: Illustration of Multi-Convs Layer within EMFormer.
  • Figure 4: Illustration of accumulative context finetuning.
  • Figure 5: Comparison of our method with 4 competitors on denormalized RMSE $\downarrow$ and ACC $\uparrow$ in 0.25° ERA5.
  • ...and 15 more figures

Theorems & Definitions (4)

  • Theorem 2.1: Efficiency and Equivalence of Multi-Conv Layer
  • Theorem 2.2: Adaptive Loss Weighting
  • Theorem 1.1: Efficiency and Equivalence of Multi-Conv Layer
  • Theorem 1.2: Adaptive Loss Weighting