Using data assimilation tools to dissect GraphDOP

TL;DR

The paper tackles the interpretability and explainability of GraphDOP, a data-driven, observation-only weather predictor, by transferring Data Assimilation sensitivity diagnostics to a Graph Neural Network framework. It uses reverse-mode automatic differentiation to analyze the Jacobian $J=\partial f/\partial x$ and related products, and applies Forecast Sensitivity-based Observation Impact (FSOI) to quantify how individual observations influence forecasts. Findings show GraphDOP develops a physically meaningful internal Earth-system representation, with learned spatial–temporal sensitivities and data-type contributions that align with atmospheric processes, including storm motion and humidity-driven advection. The work argues these diagnostics can guide training data selection, model refinement, and trust in AI-DOP forecasts, and it outlines avenues for online QC and OSE-like experiments to further improve robustness and transparency.

Abstract

The Data Assimilation (DA) community has been developing various diagnostics to understand the importance of the observing system in accurately forecasting the weather. They usually rely on the ability to compute the derivatives of the physical model output with respect to its initial condition. For example, the Forecast Sensitivity-based Observation Impact (FSOI) estimates the impact on the forecast error of each observation processed in the DA system. This paper presents how these DA diagnostic tools are transferred to Machine Learning (ML) models, as their derivatives are readily available through automatic differentiation. We specifically explore the interpretability and explainability of the observation-driven GraphDOP model developed at the European Centre for Medium-Range Weather Forecasts (ECMWF). The interpretability study demonstrates the effectiveness of GraphDOP's sliding attention window to learn the meteorological features present in the observation datasets and to learn the spatial relationships between different regions. Making these relationships more transparent confirms that GraphDOP captures real, physically meaningful processes, such as the movement of storm systems. The explainability of GraphDOP is explored by applying the FSOI tool to study the impact of the different observations on the forecast error. This inspection reveals that GraphDOP creates an internal representation of the Earth system by combining the information from conventional and satellite observations.

Figures (7)

• Figure 1: Sensitivity of input radiances (K) from NOAA-20 ATMS channel 6 to forecast 2m-temperature (C) in Kerguelen (orange dot) after 1500 (top), 9000 (middle) and 50000 (bottom) iterations in the training. This was computed for a forecast lead time of 12 hours on 01/01/2023.
• Figure 2:
• Figure 3: Relative contribution of each data type in the global forecast error after 12 hours. Negative (positive) values correspond to a decrease (increase) in the forecast error. Statistics have been averaged between 01/01/2023 and 01/03/2023.
• Figure 4: Relative contribution of ATMS channel 22 (left) and surface pressure (right) in the global forecast error after 12 hours. Negative (positive) values correspond to a decrease (increase) in the forecast error. Statistics have been averaged between 01/01/2023 and 01/03/2023.
• Figure 5: Relative contribution of each data type (columns) for different forecast error metrics (rows) after 12 hours (top panel) and 36 hours (bottom panel). Statistics have been averaged between 01/01/2023 and 01/03/2023.