XiChen: An observation-scalable fully AI-driven global weather forecasting system with 4D variational knowledge

TL;DR

XiChen introduces a fully AI-driven global weather forecasting system that scales assimilation across observation types by integrating a foundation model with 4DVar knowledge. The approach delivers a 15-second end-to-end pipeline on a single GPU and achieves 8.75+ day forecast lead times with accuracy comparable to operational NWP systems, while maintaining physical balance during DA. It demonstrates robust performance across a year-long DA cycle, 10-day forecasts, and tropical cyclone tracks, aided by a cascaded DA framework and selective fine-tuning of observation operators and DA components. While presenting a powerful complementary tool to NWP, XiChen acknowledges limitations such as smoothing tendencies, resolution constraints, and reliance on reanalysis data for training, outlining future directions toward higher resolution, more observational diversity, and probabilistic forecasting.

Abstract

Artificial intelligence (AI)-driven models have the potential to revolutionize weather forecasting, but still rely on initial conditions generated by costly Numerical Weather Prediction (NWP) systems. Although recent end-to-end forecasting models attempt to bypass NWP systems, these methods lack scalable assimilation of new types of observational data. Here, we introduce XiChen, an observation-scalable fully AI-driven global weather forecasting system, wherein the entire pipeline, from Data Assimilation (DA) to medium-range forecasting, can be accomplished within only 15 seconds. XiChen is built upon a foundation model that is pre-trained for weather forecasting and subsequently fine-tuned to serve as both observation operators and DA models, thereby enabling the scalable assimilation of conventional and raw satellite observations. Furthermore, the integration of Four-Dimensional Variational (4DVar) knowledge ensures XiChen to achieve DA and medium-range forecasting accuracy comparable to operational NWP systems, with skillful forecasting lead time beyond 8.75 days. A key feature of XiChen is its ability to maintain physical balance constraints during DA, enabling observed variables to correct unobserved ones effectively. In single-point perturbation DA experiments, XiChen exhibits flow-dependent characteristics similar to those of traditional 4DVar systems. These results demonstrate that XiChen holds strong potential for fully AI-driven weather forecasting independent of NWP systems.

Figures (9)

• Figure 1: Comparison of the average latitude RMSE of the one-year DA cycle using XiChen (red lines) as well as the operational analysis fields of GFS (yellow lines) and IFS HRES (blue lines) using testing data during 2023. The analysis encompassed 16 distinct variables, including geopotential heights at 300 hPa (Z300), 500 hPa (Z500), and 700 hPa (Z700); 2-meter temperature (T2M); temperatures at 300 hPa (T300), 500 hPa (T500), 700 hPa (T700), and 850 hPa (T850); u- and v-components of the wind at 10 meters (U10 and V10, respectively); u-components of the wind at 300 hPa (U300), 500 hPa (U500), and 700 hPa (U700); and specific humidity at 300 hPa (Q300), 500 hPa (Q500), and 700 hPa (Q700). All analysis fields are evaluated against the ERA5 reanalysis dataset.
• Figure 2: Comparison of the average latitude RMSE (first and third rows) and ACC (second and fourth rows) of the 10-day medium-range weather forecasting using XiChen (red lines) as well as the operational forecasting results of GFS (yellow lines) and IFS HRES (blue lines) using testing data during 2023. The evaluation encompasses eight meteorological variables: geopotential height at 500 hPa (Z500) and 850 hPa (Z850), 2-meter temperature (T2M), temperature at 850 hPa (T850), Mean Sea Level Pressure (MSLP), 10-meter zonal (10U) and meridional (10V) wind components, as well as specific humidity at 500 hPa (Q500). All analysis fields are evaluated against the ERA5 reanalysis dataset.
• Figure 3: Headline scorecards of the average normalized RMSEs for 10-day medium-range weather forecasting, based on the assimilation of different observations. The colors denote the percentage difference relative to the baseline, which assimilates all observations, including GDAS prepbufr, AMSU-A, MHS, ASCAT, and SATWND. Blue shading indicates a reduction in normalized RMSE, while red shading indicates an increase.
• Figure 4: Headline scorecards of the global-averaged latitude-weighted RMSEs for 10-day medium-range weather forecasting, based on the assimilation of different observations. The 10-day medium-range forecasts based on analysis fields that assimilate all observational data—including GDAS prepbufr, AMSU-A, MHS, ASCAT, and SATWND—are compared with the forecasts based on analysis fields that assimilate only the GDAS prepbufr. The colors indicate the percentage difference relative to the baseline, which assimilates only the GDAS prepbufr. Blue shading represents a reduction in RMSE, while red shading represents an increase.
• Figure 5: Changes in the analysis fields resulting from a 5 K perturbation to the AMSU-A observation at the selected location, where the background field is the ERA5 data at 00 UTC on July 24, 2023. The perturbation was introduced at 03 UTC, three hours after the analysis time, near Typhoon Doksuri at 20$^\circ$ N, and 120 $^\circ$ E (indicated by the yellow dot). The first row displays the horizontal spatial distribution of analysis changes for channels 5 to 7 at 700, 200, and 200 hPa. Shading represents temperature differences (units: $K$), while vectors indicate wind differences (units: $m s^{-1}$) resulting from the assimilation of single-point perturbation observations. The solid contour illustrates the background geopotential field (units: $m^2 s^{-2}$). The second row displays the vertical distribution along the same west-east cross-section.