Making Tunable Parameters State-Dependent in Weather and Climate Models with Reinforcement Learning

TL;DR

This paper tackles the persistent biases in weather and climate models by replacing static, offline-tuned parametrisations with state-dependent, online learned adjustments driven by reinforcement learning. It develops climateRL environments spanning SCBC, RCE, and Budyko–Sellers EBMs and evaluates nine RL algorithms in both single-agent and federated multi-agent setups, identifying TQC, DDPG, and TD3 as the most reliable across configurations. Key findings show that RL-powered parametrisations can reduce meridional biases, adjust vertical temperature structures, and adapt radiative parameters in a physically meaningful way, with federated learning enabling regional specialization and faster convergence. The results suggest RL offers a scalable pathway for online learning within numerical models, preserving conservation laws while enabling regime-aware, state-dependent parametrisations. The work supports future integration into full GCMs and operational models, highlighting the balance between local specialization and global coherence in distributed climate simulations.

Abstract

Weather and climate models rely on parametrisations to represent unresolved sub-grid processes. Traditional schemes rely on fixed coefficients that are weakly constrained and tuned offline, contributing to persistent biases that limit their ability to adapt to the underlying physics. This study presents a framework that learns components of parametrisation schemes online as a function of the evolving model state using reinforcement learning (RL) and evaluates the resulting RL-driven parameter updates across a hierarchy of idealised testbeds spanning a simple climate bias correction (SCBC), a radiative-convective equilibrium (RCE), and a zonal mean energy balance model (EBM) with both single-agent and federated multi-agent settings. Across nine RL algorithms, Truncated Quantile Critics (TQC), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed DDPG (TD3) achieved the highest skill and the most stable convergence across configurations, with performance assessed against a static baseline using area-weighted RMSE, temperature profile and pressure-level diagnostics. For the EBM, single-agent RL outperformed static parameter tuning with the strongest gains in tropical and mid-latitude bands, while federated RL on multi-agent setups enabled geographically specialised control and faster convergence, with a six-agent DDPG configuration using frequent aggregation yielding the lowest area-weighted RMSE across the tropics and mid-latitudes. The learnt corrections were also physically meaningful as agents modulated EBM radiative parameters to reduce meridional biases, adjusted RCE lapse rates to match vertical temperature errors, and stabilised SCBC heating increments to limit drift. Overall, results highlight RL to deliver skilful state-dependent, and regime-aware parametrisations, offering a scalable pathway for online learning within numerical models.

Figures (21)

• Figure 1: ebm-v0 (left) and ebm-v1 (right) single-agent setup. In ebm-v1, the global agent observes the full zonal-mean temperature profile and outputs latitude-dependent OLR parameters $\{A_\phi, B_\phi\}$ as well as independent ones $\{\alpha_0, \alpha_2, D\}$. Loss from observations are computed over all 96 latitudes.
• Figure 2: ebm-v2 multi-agent ensemble with FedRL agents operate on assigned regions with local rewards while receiving the global profile as input. Periodic aggregation every $K$ episodes synchronises policy weights across $n$ agents.
• Figure 3: Developmental progression of RL algorithms, evaluated in this study. The single agent climateRL experiments span classical policy-gradient methods such as REINFORCE, on-policy algorithms --- DPG, TRPO, PPO, AVG, and advanced off-policy actor–critic approaches --- DDPG, TD3, SAC, TQC. Adapted from https://master-dac.isir.upmc.fr/rl/12_sac.pdf.
• Figure 4: Schematic of the experimental workflow for the single agent climateRL experiments (e.g., ebm-v0/1). The process begins with hyperparameter tuning on seed 1, followed by evaluation across 10 random seeds and evaluation metric computation (steps-to-threshold, variance-after-threshold, and final return difference) for all nine algorithms. Top-3 algorithms are then selected and training curves analysed with 95% confidence intervals.
• Figure 5: Pipeline for the ebm-v1/2/3 experiments. The process begins with configuring the Budyko–Sellers EBM in either single-agent (ebm-v1) or spatially decomposed multi-agent forms (ebm-v2, ebm-v3) using two (a2) or six (a6) regions. Agents are trained with one of three RL algorithms (DDPG, TD3, TQC) under FedRL coordination schemes fed05, fed10, or nofed. In multi-agent settings, policies are periodically aggregated via FedRL every $K$ episodes. Hyperparameters tuned for ebm-v1 are transferred over to ebm-v2/v3. Finally trained models are assessed on their training curves and benchmarked against a static climlab baseline, using a skill measure such as areaWRMSE across 30° latitude groups.