Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness of AI-based weather forecasts in a changing climate

Thomas Rackow, Nikolay Koldunov, Christian Lessig, Irina Sandu, Mihai Alexe, Matthew Chantry, Mariana Clare, Jesper Dramsch, Florian Pappenberger, Xabier Pedruzo-Bagazgoitia, Steffen Tietsche, Thomas Jung

TL;DR

It is shown that current state-of-the-art machine learning models trained for weather forecasting in present-day climate produce skillful forecasts across different climate states corresponding to pre-industrial, present-day, and future 2.9K warmer climates, indicating that the dynamics shaping the weather on short timescales may not differ fundamentally in a changing climate.

Abstract

Data-driven machine learning models for weather forecasting have made transformational progress in the last 1-2 years, with state-of-the-art ones now outperforming the best physics-based models for a wide range of skill scores. Given the strong links between weather and climate modelling, this raises the question whether machine learning models could also revolutionize climate science, for example by informing mitigation and adaptation to climate change or to generate larger ensembles for more robust uncertainty estimates. Here, we show that current state-of-the-art machine learning models trained for weather forecasting in present-day climate produce skillful forecasts across different climate states corresponding to pre-industrial, present-day, and future 2.9K warmer climates. This indicates that the dynamics shaping the weather on short timescales may not differ fundamentally in a changing climate. It also demonstrates out-of-distribution generalization capabilities of the machine learning models that are a critical prerequisite for climate applications. Nonetheless, two of the models show a global-mean cold bias in the forecasts for the future warmer climate state, i.e. they drift towards the colder present-day climate they have been trained for. A similar result is obtained for the pre-industrial case where two out of three models show a warming. We discuss possible remedies for these biases and analyze their spatial distribution, revealing complex warming and cooling patterns that are partly related to missing ocean-sea ice and land surface information in the training data. Despite these current limitations, our results suggest that data-driven machine learning models will provide powerful tools for climate science and transform established approaches by complementing conventional physics-based models.

Robustness of AI-based weather forecasts in a changing climate

TL;DR

It is shown that current state-of-the-art machine learning models trained for weather forecasting in present-day climate produce skillful forecasts across different climate states corresponding to pre-industrial, present-day, and future 2.9K warmer climates, indicating that the dynamics shaping the weather on short timescales may not differ fundamentally in a changing climate.

Abstract

Data-driven machine learning models for weather forecasting have made transformational progress in the last 1-2 years, with state-of-the-art ones now outperforming the best physics-based models for a wide range of skill scores. Given the strong links between weather and climate modelling, this raises the question whether machine learning models could also revolutionize climate science, for example by informing mitigation and adaptation to climate change or to generate larger ensembles for more robust uncertainty estimates. Here, we show that current state-of-the-art machine learning models trained for weather forecasting in present-day climate produce skillful forecasts across different climate states corresponding to pre-industrial, present-day, and future 2.9K warmer climates. This indicates that the dynamics shaping the weather on short timescales may not differ fundamentally in a changing climate. It also demonstrates out-of-distribution generalization capabilities of the machine learning models that are a critical prerequisite for climate applications. Nonetheless, two of the models show a global-mean cold bias in the forecasts for the future warmer climate state, i.e. they drift towards the colder present-day climate they have been trained for. A similar result is obtained for the pre-industrial case where two out of three models show a warming. We discuss possible remedies for these biases and analyze their spatial distribution, revealing complex warming and cooling patterns that are partly related to missing ocean-sea ice and land surface information in the training data. Despite these current limitations, our results suggest that data-driven machine learning models will provide powerful tools for climate science and transform established approaches by complementing conventional physics-based models.
Paper Structure (15 sections, 2 equations, 4 figures)

This paper contains 15 sections, 2 equations, 4 figures.

Table of Contents

  1. Main
  2. Data-driven weather forecasts in colder and warmer climate conditions
  3. Discussion
  4. Methods
  5. Acknowledgments
  6. Supplementary Material

Figures (4)

  • Figure 1: Skill of data-driven 10-day weather forecasts for 2m temperature in conditions representing pre-industrial, present-day, and future +2.9K climate states, for AIFS, GraphCast, and Pangu-Weather. Root-mean square error (RMSE, in K) for all models, weighted by the cosine of latitude in a, 1955, a proxy-year for pre-industrial times, b, in 2023, representing present-day climate, and c, in 2049, the +2.9K warmer world. d, Mean bias [K] for all models in 1955, e, in 2023, and f, in 2049. In 1955, Pangu-Weather cools further while the other two models show a warming. In 2023, GraphCast and AIFS show very little bias, similar to ECMWF's operational IFS forecast after 10 days, while Pangu-Weather consistently cools. In 2049, positive and negative biases for GraphCast balance while the other two models are characterised by a cold bias. All RMSE and bias evolutions in 1955, 2023, and 2049 are averaged over the daily forecasts (see Methods).
  • Figure 2: Global-mean 2m temperature evolution from daily 10-day weather forecasts under conditions representing pre-industrial (1955 as proxy), present-day (2023), and future +2.9K climate states (2049), for AIFS, GraphCast, and Pangu-Weather. All models were trained with ERA5 reanalysis from 1979-2017 (with eventual fine-tuning), and the global-mean temperature for 1979--2020 is shown (light grey lines) for context. Black dashed lines are global-mean temperatures for 1955 (ERA5), 2023 (operational analysis), and for 2049 (physics-based scenario simulation), representing initial conditions for the data-driven forecast models. For every day in the three chosen years, 10-day forecasts were performed with every model starting from 12:00 UTC, resulting in 9 forecast datasets. The global-mean temperature of these 9 forecast datasets are shown as thin (hair-like) colored lines. For 1955, a warming of $0.3\,\mathrm{K}$ is found over 10 days for GraphCast and AIFS, while Pangu-Weather cools down. For 2023, none of the models shows any clear systematic bias. In 2049, AIFS and in particular Pangu-Weather cool substantially towards the climate conditions of the training period, at a rate of $-0.04\,\mathrm{K/day}$ and $-0.07\,\mathrm{K/day}$, respectively. Interestingly, GraphCast follows the climate projection in the global mean closely, but this is achieved by a compensation of cooling over land with a warming over the ocean.
  • Figure 3: Mean 2m temperature drift over 10 days in 2049 from the data-driven models when compared to the reference physics-based simulation.a, for AIFS v0.2.1, b, GraphCast, and c, for Pangu-Weather. d, Global 2m temperature difference between the present-day and future year used in this study as references (2023-2049, operational analysis minus scenario). Annual-mean fields for the data-driven models were constructed by combining 365 daily means from the end of their individual 10-day forecasts. The annual-mean from the reference simulation was then subtracted. There are many coherent areas with the same color in panels a-c and panel d where the models' drift towards the present-day conditions they were trained for aligns with the greatest differences in 2m temperature between future and present-day conditions. Note the different color range in panel d compared to the other panels.
  • Figure S1: Mean 2m temperature drift over 10 days in 2023 (left column) and 1955 (right column) from the data-driven models (and operational IFS forecasts in 2023) when compared to their reference. References are the operational analysis in 2023 and ERA5 back extension data in 1955. a) Operational IFS forecasts (HRES), b,c) AIFS v0.2.1, d,e) GraphCast, and f,g) Pangu-Weather. Annual-mean fields for the data-driven models were constructed by combining 365 daily means from the end of the individual 10-day forecasts into an annual dataset, and the annual-means from the references were then subtracted.