Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

Shengchao Chen, Guodong Long, Jing Jiang, Chengqi Zhang

TL;DR

The paper presents LM-Weather, a framework that repurposes pre-trained language models as foundation models for on-device meteorological variable modeling. By integrating lightweight Task Adapters that decompose weather sequences into Trend, Seasonal, and Residual components, and coupling this with Parameter Adapters and LoRA-based low-rank updates, the approach achieves personalized, efficient on-device forecasting and imputation. Real-world datasets (ODW1/ODW2) and federated-like experimentation demonstrate large performance gains over state-of-the-art baselines, strong few-shot and zero-shot generalization, and substantial reductions in communication overhead while preserving privacy. The work highlights the potential of weather foundation modeling on devices and suggests avenues for extending to multimodal, region-specific, and privacy-preserving distributed weather analytics.

Abstract

This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

TL;DR

The paper presents LM-Weather, a framework that repurposes pre-trained language models as foundation models for on-device meteorological variable modeling. By integrating lightweight Task Adapters that decompose weather sequences into Trend, Seasonal, and Residual components, and coupling this with Parameter Adapters and LoRA-based low-rank updates, the approach achieves personalized, efficient on-device forecasting and imputation. Real-world datasets (ODW1/ODW2) and federated-like experimentation demonstrate large performance gains over state-of-the-art baselines, strong few-shot and zero-shot generalization, and substantial reductions in communication overhead while preserving privacy. The work highlights the potential of weather foundation modeling on devices and suggests avenues for extending to multimodal, region-specific, and privacy-preserving distributed weather analytics.

Abstract

This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.
Paper Structure (84 sections, 4 theorems, 16 equations, 5 figures, 55 tables)

This paper contains 84 sections, 4 theorems, 16 equations, 5 figures, 55 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. On-device Meteorological Variable (Sequence) Modeling
  4. Problem Formulation.
  5. Language Models in Time Series
  6. Taming PLMs for On-device Meteorological Variable (Sequence) Modeling
  7. Overview.
  8. Local Training
  9. Task Adapter.
  10. Lightweight Operations.
  11. Parameter Adapter.
  12. Communication
  13. Theorems
  14. Experiments
  15. Datasets
  16. ...and 69 more sections

Key Result

Theorem 4.1

Given a weather series ${\mathcal{X}} = {\mathcal{X}}_{\text{Trend}, t} + {\mathcal{X}}_{\text{Seasonal}, t} + {\mathcal{X}}_{\text{Residual}, t}$, $t \in [t_1, t_n ]$. Let ${\bm{E}} = \{e_1, e_2, ..., e_n\}$ denotes a set of orthogonal bases. Lets ${\bm{E}}_{\text{Seasonal}} \subseteq {\bm{E}}$ den

Figures (5)

  • Figure 1: Overview. (a) Schematic of LM-Weather, each client using personalized adapter to endow the PLM for local weather awareness, only low-rank matrices are transmitted to enhance efficiency during communication; (b) Brief structure of PLM on each client, detailed architecture can be found in Appendix; (c) Task Adapter, the multivariate weather series input splits into two paths. The first path isolates the trend, seasonal, and residual elements, which each go through independent generator to produce specific adapters; (d) Architecture of the generator for each decomposed element; (e) Schematic diagram of Channel-Independent Patching nie2022time.
  • Figure 2: Visualisation of partial variables in ODW1 dataset, where we have selected the first 1,000 time points for presentation. The data distribution from different ground weather stations exhibit significant heterogeneity, and even though the trends of some variables may be similar, there are serious differences in magnitudes. The selected variables are, from left to right, temperature, precipitation in 1-hour/12-hour, humidity, and wind direction.
  • Figure 3: Visualisation of partial variables in ODW2 dataset, where we have selected the first 1,000 time points for presentation. The data distribution from different ground weather stations exhibit significant heterogeneity, and even though the trends of some variables may be similar, there are serious differences in magnitudes. The selected variables are, from left to right, humidity, precipitation, temperature, wind direction, and wind speed.
  • Figure 4: (Figure \ref{['fig:data2_vis_1st']} continued) Visualisation of partial variables in ODW2 dataset, where we have selected the first 1,000 time points for presentation. The selected variables are, from left to right, humidity, precipitation, temperature, wind direction, and wind speed.
  • Figure 5: Schematic diagram of the PLM in LM-Weather, where we introduced LoRA to the PLM, to achieve more reliable cross-domain knowledge transfer while at the same time ensuring greater efficiency in adapting to low-resource weather devices.

Theorems & Definitions (6)

  • Theorem 4.1: Decomposition Rationality from Time Series
  • Theorem 4.2: Exchange Low-Rank Matrices Ensures Privacy
  • Theorem C.1: Decomposition Rationality from Time Series
  • proof
  • Theorem C.2: Exchange Low-Rank Matrices Ensures Privacy
  • proof