ADAF: An Artificial Intelligence Data Assimilation Framework for Weather Forecasting

TL;DR

An artificial intelligence-based data assimilation framework (ADAF) to generate high-quality kilometer-scale analysis and is shown to be efficient and effective in real-world DA, underscoring its potential role in operational weather forecasting.

Abstract

The forecasting skill of numerical weather prediction (NWP) models critically depends on the accurate initial conditions, also known as analysis, provided by data assimilation (DA). Traditional DA methods often face a trade-off between computational cost and accuracy due to complex linear algebra computations and the high dimensionality of the model, especially in nonlinear systems. Moreover, processing massive data in real-time requires substantial computational resources. To address this, we introduce an artificial intelligence-based data assimilation framework (ADAF) to generate high-quality kilometer-scale analysis. This study is the pioneering work using real-world observations from varied locations and multiple sources to verify the AI method's efficacy in DA, including sparse surface weather observations and satellite imagery. We implemented ADAF for four near-surface variables in the Contiguous United States (CONUS). The results indicate that ADAF surpasses the High Resolution Rapid Refresh Data Assimilation System (HRRRDAS) in accuracy by 16% to 33% for near-surface atmospheric conditions, aligning more closely with actual observations, and can effectively reconstruct extreme events, such as tropical cyclone wind fields. Sensitivity experiments reveal that ADAF can generate high-quality analysis even with low-accuracy backgrounds and extremely sparse surface observations. ADAF can assimilate massive observations within a three-hour window at low computational cost, taking about two seconds on an AMD MI200 graphics processing unit (GPU). ADAF has been shown to be efficient and effective in real-world DA, underscoring its potential role in operational weather forecasting.

Figures (15)

• Figure 1: An overview of ADAF and the architecture of neural networks used in ADAF. (a) An overview of ADAF. The input consists of three types of data, including observations within 3-hour window, background, and topography. The output is the analysis. (b) The neural network architecture implemented in ADAF consists of an encoder, a decoder, and a reconstruction module. (c) Details of the decoder, which includes residual Swin Transformer blocks (RSTB), Swin Transformer layers (STL), and Multi-head Self Attention (MSA).
• Figure 2: Domain-averaged root mean squared errors (RMSE) and correlation (CORR) for ADAF analysis (denoted as ADAF), HRRRDAS analysis (denoted as HRRRDAS) and RTMA for four near-surface variables: T2M, Q, U10 and V10. The evaluation is performed by comparing with withheld surface weather observations. The results indicate that ADAF analysis have better accuracy with lower RMSE and higher CORR than HRRRDAS analysis. ADAF analysis aligns more closely with observations than HRRRDAS. The mean RMSE values of HRRRDAS analysis for T2M, Q, U10, and V10 are 1.81, 1.21, 1.96, and 1.98. The ADAF analysis shows lower RMSE value: 1.67, 1.14, 1.81, and 1.83, showing improvements of 7.7%, 5.7%, 7.7%, and 7.6%.
• Figure 3: Domain-averaged Root Mean Squared Errors (RMSE) and Correlation (CORR) for ADAF analysis (denoted as ADAF) and HRRRDAS analysis (denoted as HRRRDAS) for four near-surface variables: T2M, Q, U10, and V10. The evaluation is performed by comparing with RTMA. 'Land' indicates that metrics are computed only over land. The results indicate that ADAF analysis surpasses HRRRDAS analysis in depicting surface atmospheric conditions. The decrease in errors over land compared to the full study area is attributed to ADAF's efficient assimilation of surface observations. Additionally, the error variability in ADAF analysis is less than that in HRRRDAS analysis, underscoring the robustness of ADAF.
• Figure 4: The Mean Absolute Errors (MAE) for ADAF and HRRRDAS analysis, compared with RTMA. The first and second columns present the MAE spatial distribution for ADAF and HRRRDAS analysis, respectively. The third column illustrates the ratio of MAE reduction, calculated as $(MAE_{\text{HRRRDAS}} - MAE_{\text{ADAF}}) / MAE_{\text{HRRRDAS}}$. In most areas, ADAF analysis generally shows lower MAE compared to HRRRDAS analysis. About 82%, 86%, 93%, and 92% of the areas show improvements, with average MAE reduction of 16%, 26%, 35%, and 33% for T2M, Q, U10, and V10, respectively.
• Figure 5: Relationship between analysis errors and the sparsity of observations, which is defined by the observation grid point (see Equation \ref{['eq:obs_grid_ratio']}). The errors, represented as the domain-averaged RMSE and CORR, in comparison with RTMA. The ADAF analysis (depicted with blue lines) shows lower RMSE and higher CORR compared to the HRRRDAS analysis (depicted with red lines) at various observation sparsity levels. With the reduction in surface observations, the error grows, implying that a greater number of observations can improve the accuracy of the analysis. The results demonstrate that ADAF is robust to generating high-quality analysis even when observations are extremely sparse (with an observation grid ratio of 0.5%).