Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

TL;DR

The paper tackles the challenge of long-range weather forecasting by replacing autoregressive rollouts with a single-timestep probabilistic model trained on a vast synthetic dataset generated by a short-timestep teacher (DLESyM). It introduces long-range distillation using a HEALPix diffusion-based student, calibrated with classifier-free guidance, and demonstrates that forecast skill scales with synthetic data volume and improves after ERA5 fine-tuning. The approach yields performance competitive with operational ECMWF S2S forecasts in real-world tests and provides a scalable pathway to harness AI-generated climate simulations for long-range prediction. This work broadens the design space for AI weather models by showing how synthetic data from autoregressive emulators can bootstrap high-fidelity long-range forecasts.

Abstract

Accurate long-range weather forecasting remains a major challenge for AI models, both because errors accumulate over autoregressive rollouts and because reanalysis datasets used for training offer a limited sample of the slow modes of climate variability underpinning predictability. Most AI weather models are autoregressive, producing short lead forecasts that must be repeatedly applied to reach subseasonal-to-seasonal (S2S) or seasonal lead times, often resulting in instability and calibration issues. Long-timestep probabilistic models that generate long-range forecasts in a single step offer an attractive alternative, but training on the 40-year reanalysis record leads to overfitting, suggesting orders of magnitude more training data are required. We introduce long-range distillation, a method that trains a long-timestep probabilistic "student" model to forecast directly at long-range using a huge synthetic training dataset generated by a short-timestep autoregressive "teacher" model. Using the Deep Learning Earth System Model (DLESyM) as the teacher, we generate over 10,000 years of simulated climate to train distilled student models for forecasting across a range of timescales. In perfect-model experiments, the distilled models outperform climatology and approach the skill of their autoregressive teacher while replacing hundreds of autoregressive steps with a single timestep. In the real world, they achieve S2S forecast skill comparable to the ECMWF ensemble forecast after ERA5 fine-tuning. The skill of our distilled models scales with increasing synthetic training data, even when that data is orders of magnitude larger than ERA5. This represents the first demonstration that AI-generated synthetic training data can be used to scale long-range forecast skill.

Figures (10)

• Figure 1: Schematic of long-range distillation approach. First, an autoregressive teacher model is trained for short lead time prediction using reanalysis datasets like ERA5 --- in our case we leverage DLESyM as a pre-trained autoregressive teacher capable of stable long-running simulations. The autoregressive teacher is used to generate a huge simulation with orders of magnitude higher temporal coverage than ERA5. Finally, a long timestep probabilistic model, the "student", is trained on the autoregressive teacher to do long-range prediction in a single model timestep.
• Figure 2: Error growth in DLESyM autoregressive ensemble members for varying initial condition perturbation strengths. Global average $Z_{500}$ RMSE of DLESyM ensemble members with respect to a withheld ensemble member for varying initial condition perturbation sizes (colored lines) and the real-world global average $Z_{500}$ RMSE of IFS for reference (black dashed line). The initial condition condition perturbation strengths in the legend are provided in normalized units since we perturb all fields using a multiple of each field's global standard deviation. For $Z_{500}$, the standard deviation is 2605 $m^2/s^2$.
• Figure 3: Example distilled student model 2 m air temperature forecasts across a range of lead times initialized on 2017-01-01. Ground truth forecast targets from the withheld DLESyM evaluation simulation or (a) medium-range, (b) S2S, and (c) seasonal lead times. (d-f) Single forecast member predictions for each lead time. Both forecast targets and ground truth targets are visualized as anomalies from climatology (a-f). (g-i) Ensemble mean forecast error for a 32-member ensemble forecast for each lead time. Medium-range forecasts (a, d, g) are daily means at 7-day lead time, S2S forecasts (b, e, h) are weekly means at 4-week lead time, and seasonal forecasts (c, f, i) are 4-week means at 12-week lead time.
• Figure 4: Scaling of distilled student model S2S forecast skill with synthetic training dataset size. (a) Elucidated diffusion model (EDM) loss karras2022elucidating throughout training for a 40 year training dataset (blue) and the full 11,000 year dataset (orange). Dashed lines indicate training loss and solid the validation loss. Learning curves have been smoothed for plotting purposes using bin averaging. (b) Distilled model week 4 2 m air temperature global mean CRPS with increasing number of training years (orange) and the corresponding minimum validation EDM loss achieved during training (blue). Dashed lines indicate the length of the ERA5 record and the CRPS skill of climatology for reference. Note all axes are logarithmic.
• Figure 5: Calibration of medium-range distilled student model forecasts using classifier-free guidance. (a) Day 7 global $Z_{500}$ ensemble spread for varying classifier-free guidance strengths (black solid) with reference ensemble spread for a 20 year climatology (red dashed). (b) Distilled model ensemble spread-skill ratio for varying classifier-free guidance strengths (black solid). Ensemble spread-skill plot uses unbiased estimator for ensemble spread. (c) Day 7 global mean $Z_{500}$ CRPS for distilled student model with varying classifier-free guidance strengths (black solid), compared to a probabilistic (red dashed) and deterministic (blue dashed) 20 year climatology. Vertical grid lines align with integer guidance strengths, with an extra line at 0.5 between 0 and 1.