Global 3D Reconstruction of Clouds & Tropical Cyclones

TL;DR

This work tackles the challenge of forecasting tropical cyclones by recovering vertically resolved cloud structure from satellite observations. It introduces SWinSatMAE, a pre-training–fine-tuning framework that fuses data from multiple geostationary satellites to map 2D radiance into 3D fields of cloud properties $Z$, $\mathrm{IWC}$, and $\mathrm{r_e}$. The approach delivers global near real-time 3D reconstructions of clouds and tropical cyclone structure, outperforming a 2D U-Net baseline and providing robust predictions across cloud types and high viewing angles, including during intense events like Dorian. This framework broadens observational capabilities, offering new inputs for understanding TC intensification and improving forecasts, with future work on edge sharpening, comprehensive validation, and sensor-agnostic generalization to integrate with operational forecasting systems.

Abstract

Accurate forecasting of tropical cyclones (TCs) remains challenging due to limited satellite observations probing TC structure and difficulties in resolving cloud properties involved in TC intensification. Recent research has demonstrated the capabilities of machine learning methods for 3D cloud reconstruction from satellite observations. However, existing approaches have been restricted to regions where TCs are uncommon, and are poorly validated for intense storms. We introduce a new framework, based on a pre-training--fine-tuning pipeline, that learns from multiple satellites with global coverage to translate 2D satellite imagery into 3D cloud maps of relevant cloud properties. We apply our model to a custom-built TC dataset to evaluate performance in the most challenging and relevant conditions. We show that we can - for the first time - create global instantaneous 3D cloud maps and accurately reconstruct the 3D structure of intense storms. Our model not only extends available satellite observations but also provides estimates when observations are missing entirely. This is crucial for advancing our understanding of TC intensification and improving forecasts.

Figures (11)

• Figure 1: Overview of our ML pipeline. We select the 11 closest-matched spectral channels from the 16 channels of GOES and Himawari to create a consistent model input. During pre-training, the image encoder and decoder learn cloud structures by reconstructing masked images. During fine-tuning a 3D decoder is trained using paired image-profile pairs. Multiple prediction heads are used to predict different cloud properties simultaneously.
• Figure 2: 3D reconstructions of (a) Z, (b) IWC, and (c) $\mathrm{r_e}$ by the SWinSatMAE model for TC Dorian. The geostationary image from GOES channel 7 is shown under each 3D render, with the location of the CloudSat track marked in red. For validation purposes, the SWinSatMAE predictions along the CloudSat overpass are compared to the CloudSat retrievals and to the multi-variable U-Net baseline.
• Figure 3: Spatial RMSE distribution of Z (top), IWC (middle), and $\mathrm{r_e}$ (bottom) predictions for the baseline model (left) and the SWinSatMAE model (middle), along with their difference (right).
• Figure 4: Comparison of masked, predicted, and original images (MSG example) during pre-training of our SWinSatMAE model. We pretrained our model for 50 epochs.
• Figure 5: Comparison of prediction performance across the globe between U-Net models trained on our three satellites, MSG, GOES and Himawari individually (top), and a U-Net model trained on all three satellites together (Multi-Satellite U-Net, middle).