"What is a realistic forecast?" Assessing data-driven weather forecasts, a journey from verification to falsification

TL;DR

The paper addresses the challenge of defining and assessing realism in data-driven weather forecasts, going beyond traditional verification metrics. It formalizes three realism types—$Type\ 1$ functional realism with $V = v(x_i,y_i)$, $Type\ 2$ structural realism with $D = d(X,Y)$, and $Type\ 3$ physical realism using falsification $F = f(X_i,\mathcal{K})$—and proposes a falsification-based framework that complements verification and diagnostics. It discusses how these types relate, the role of interpretability and the knowledge base in building trust, and outlines a practical evaluation journey (verification, diagnostics, falsification) with attention to issues like hallucinations. The framework aims to guide the development and assessment of data-driven forecasts in operational contexts by providing a structured, knowledge-grounded approach to determine forecast realism and usefulness.

Abstract

The artificial intelligence revolution is fueling a paradigm shift in weather forecasting: forecasts are generated with machine learning models trained on large datasets rather than with physics-based numerical models that solve partial differential equations. This new approach proved successful in improving forecast performance as measured with standard verification metrics such as the root mean squared error. At the same time, the realism of data-driven weather forecasts is often questioned and considered as an Achilles' heel of machine learning models. How 'forecast realism' can be defined and how this forecast attribute can be assessed are the two questions simultaneously addressed here. Inspired by the seminal work of Murphy (1993) on the definition of 'forecast goodness', we identify 3 types of realism and discuss methodological paths for their assessment. In this framework, falsification arises as a complementary process to verification and diagnostics when assessing data-driven weather models.

Figures (2)

• Figure 1: Schematic of the two types of logic applied in weather forecasting: A) deduction and B) induction. In A), the rules of the numerical model are derived from the laws of physics for theory-driven models or as an output of B) for data-driven models.
• Figure 2: Illustration of evaluation activities to assess the 3 types of realism discussed in this manuscript: A) verification $V$ to assess the functional realism, B) diagnostic $D$ to assess the structural realism, and C) falsification based on knowledge base $\mathcal{K}$ to check for physical realism.