Exploring Design Choices for Autoregressive Deep Learning Climate Models

TL;DR

This work tackles the challenge of achieving physically plausible long-term rollouts for autoregressive deep learning climate models, addressing decadal timescales rather than short-term forecasts. It quantitatively compares three architectures—FourCastNet, SFNO, and ClimaX—on ERA5-based WeatherBench data at a coarse 5.625° grid, systematically varying autoregressive steps, model capacity, and prognostic variables. The key findings show that SFNO is relatively robust to hyperparameters, while multi-step training improves long-horizon stability across architectures; however, instability can still arise from random seeds and variable selections. The results offer practical guidance for building data-driven climate simulators and highlight future directions, including incorporating external forcings and ocean coupling to enable climate-scale projections.

Abstract

Deep Learning models have achieved state-of-the-art performance in medium-range weather prediction but often fail to maintain physically consistent rollouts beyond 14 days. In contrast, a few atmospheric models demonstrate stability over decades, though the key design choices enabling this remain unclear. This study quantitatively compares the long-term stability of three prominent DL-MWP architectures - FourCastNet, SFNO, and ClimaX - trained on ERA5 reanalysis data at 5.625° resolution. We systematically assess the impact of autoregressive training steps, model capacity, and choice of prognostic variables, identifying configurations that enable stable 10-year rollouts while preserving the statistical properties of the reference dataset. Notably, rollouts with SFNO exhibit the greatest robustness to hyperparameter choices, yet all models can experience instability depending on the random seed and the set of prognostic variables

Figures (9)

• Figure 1: Area-weighted normalized RMSE scores averaged over the variables tas, uas, vas, ta850 and zg500 for model configurations $0-8$ with layer size $L=4$. The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis.
• Figure 2: As \ref{['fig:overview-layers4']} with $L=6$: Area-weighted normalized RMSE scores averaged over the variables tas, uas, vas, ta850 and zg500 for model configurations $0-8$ with layer size $L=6$. The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis.
• Figure 3: As \ref{['fig:overview-layers4']} with $L=8$: Area-weighted normalized RMSE scores averaged over the variables tas, uas, vas, ta850 and zg500 for model configurations $0-8$ with layer size $L=8$. The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis.
• Figure 4: As \ref{['fig:overview-layers4']}, but temporal standard deviations are compared instead of temporal means: Area-weighted normalized RMSE scores for the temporal standard deviation averaged over the variables tas, uas, vas, ta850 and zg500 for model configurations $0-8$ with layer size $L=4$. The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis.
• Figure 5: As \ref{['fig:overview-layers4']} averaged over all 33 variables. Area-weighted normalized RMSE scores averaged over all 33 prognostic variables for model configurations $0-8$ with layer size $L=4$. The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis.