Decision-oriented benchmarking to transform AI weather forecast access: Application to the Indian monsoon

TL;DR

The paper tackles the gap between meteorological benchmarking and real-world decision support by introducing a decision-oriented operational benchmarking framework for AI-based weather predictions, demonstrated on Indian monsoon onset. Six open-source AIWP models plus one NWP model are evaluated against IMD ground truth using both deterministic and probabilistic metrics, with hindcasts spanning pre- and post-satellite eras to address small sample sizes. The framework shows that local onset forecasts can achieve skill up to about 15 days deterministically and extend probabilistic skill to roughly 20–30 days for several models; it also highlights reliability issues and the value of multi-model ensembles. The framework informed a 2025 government dissemination of calibrated AI-based onset forecasts to 38 million farmers, illustrating a practical path from benchmarking to large-scale, impact-oriented deployment in LMICs and offering a blueprint for future similar efforts.

Abstract

Artificial intelligence weather prediction (AIWP) models now often outperform traditional physics-based models on common metrics while requiring orders-of-magnitude less computing resources and time. Open-access AIWP models thus hold promise as transformational tools for helping low- and middle-income populations make decisions in the face of high-impact weather shocks. Yet, current approaches to evaluating AIWP models focus mainly on aggregated meteorological metrics without considering local stakeholders' needs in decision-oriented, operational frameworks. Here, we introduce such a framework that connects meteorology, AI, and social sciences. As an example, we apply it to the 150-year-old problem of Indian monsoon forecasting, focusing on benefits to rain-fed agriculture, which is highly susceptible to climate change. AIWP models skillfully predict an agriculturally relevant onset index at regional scales weeks in advance when evaluated out-of-sample using deterministic and probabilistic metrics. This framework informed a government-led effort in 2025 to send 38 million Indian farmers AI-based monsoon onset forecasts, which captured an unusual weeks-long pause in monsoon progression. This decision-oriented benchmarking framework provides a key component of a blueprint for harnessing the power of AIWP models to help large vulnerable populations adapt to weather shocks in the face of climate variability and change.

Figures (9)

• Figure 1: Decision-oriented operational benchmarking framework: the Indian monsoon onset example. Panels (a)–(b) present an operationally oriented benchmarking framework for global AIWP models using locally relevant, decision-oriented metrics. One example is an agriculturally relevant metric of the Indian summer monsoon onset, which marks the beginning of the primary rainy season over India. As shown in (a), the timing of the onset (marked by a red line), followed by continuous rainfall, is relevant to planting decisions and, consequently, for achieving good crop yields. Panel (b) outlines the key steps involved in the operational benchmarking of monsoon onset forecasts. 35-day forecasts from a suite of NWP and AIWP models are generated or retrieved for analysis. To expose the forecasts to false alarms and onset misses, in an operationally oriented setting, twice-weekly initializations from early May are used, and the forecasts are evaluated using both deterministic and probabilistic metrics against IMD gridded rain-gauge observations (rather than ERA5, which was used for training). This framework assesses model skill in capturing local monsoon onset over $\approx$ 400 km regions across India. The analysis is conducted across multiple time periods, both including and excluding the AI models’ training years, to address the small test sample size. This framework informed model selection for a large-scale dissemination of AI-based forecasts at the medium-range and subseasonal timescale to 38 million farmers in 2025 vallangimonsoon25. Panel (c) shows that the AI forecasts captured the early onset and the pause in the monsoon progression. Multi-model, probabilistic forecasts based on these models aitken2026monsoonhybrid provided timely warnings to farmers.
• Figure 1: Models' deterministic skill compared to climatology in 5-day forecast windows. Difference of CMZ-averaged MAE, FAR, and MR of onset forecasts and climatology within 5-day forecast windows for the recent test period 2019-2024. Negative values (blue) mean better than climatology; positive values (red) mean worse than climatology. For computing FAR, the classification of an onset forecast being true positive or false positive is based on tolerance of $\pm2$, $\pm2$, $\pm3$, $\pm3$, $\pm5$, and $\pm5$ days for 1-5, 6-10, 11-15, 16-20, 21-25, and 26-30 day forecast windows, respectively. $*$ and $\dagger$ have the same meaning as in Figure \ref{['fig2']}.
• Figure 2: Deterministic forecast skill of the models in the medium-range (1-15 day) and subseasonal (16-30 day) timescales across two analysis periods. (a) Forecast skill of local rainfall-based monsoon onset (defined in Methods), averaged over the CMZ (outlined in panel (b)) at 1-15 day (left) and 16-30 day (right) lead times. Horizontal bars and vertical lines show model forecasts and climatological baseline, respectively, for the recent test period (2019-2024). Error bars and shading represent the standard error for the forecasts’ and climatology’s MAE. Models with $*$ next to them have some years unavailable in this period; models with $\dagger$ include some of their training or fine-tuning years within this period (see Extended Data Table 1). Circles show results for the extended period (1965–1978; 2019–2024), with climatological baseline scores placed on the horizontal axes. (b) The 1–15 day forecast MAE (in days) of climatology, IFS, and NGCM for the recent test period is shown in each $4^\circ \times 4^\circ$ box. Shading shows the MAE difference between IFS/NGCM and climatology in the middle and right panels (blue indicates improved skill). Average MAE, FAR, and MR over the CMZ are also shown for each model. (c) Same as (a), but for a large-scale circulation-based monsoon onset defined using the Webster-Yang index (WYI) webster1992monsoon (see Methods). Spatial maps similar to (b) for a 16-30 day forecast window and averaged scores for the common model period (2004-2021) are presented in Extended Data Figure 2.
• Figure 2: Deterministic forecast skill of the models in the medium-range (1-15 day) and subseasonal (16-30 day) timescales. a) and c): Same as Figure 2 but with the square markers showing the common period (20042021), which includes AI models' training years. b): Same as Figure 2 but for the subseasonal time scale in the recent test period (2019-2024).
• Figure 3: Probabilistic performance of the models over the CMZ. (a) Brier skill score (BSS) and (b) area under the receiver operating characteristic curve (AUC) for 5-day binned forecasts over the 10 grid cells of the CMZ for the recent test period (2019–2024) (see Methods for details). BSS is computed relative to a climatological forecast, with positive values indicating improved performance compared to climatology. For AUC, higher values mean better performance. Panels (c) and (d) show aggregated probabilistic scores for the 1–15- and 1–30-day forecasts, respectively. These scores are obtained by binning the onset forecasts into 5-day windows. To account for differing numbers of ensemble members, the fair BSS and ranked probability skill score (RPSS) are used (Methods). Climatological baselines for the respective metrics during this period are shown as vertical lines. Scores for the common period (2004–2021) are indicated by square markers, with climatological scores placed on the horizontal axes (GenCast's forecasts for this period were not produced due to computational constraints). (e) Reliability diagram for onset forecasts from four probabilistic models for the 1–15-day forecast window for the recent test period. Confidence intervals of two standard errors around the observed frequencies are shown. Models marked with $*$ have year(s) unavailable during this period. More Results from (a)-(e) but for the common period are presented in Extended Data Figure 3.