Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution

TL;DR

Cloud4D addresses the observational gap in high-resolution cloud state estimation by reconstructing a four-dimensional cloud field from synchronized ground-based cameras. It introduces a homography-guided 2D-to-3D transformer to estimate a high-resolution $LWC$ field on a grid of size $N_x \times N_y \times N_z$ with voxel size $25~\mathrm{m}$ and cadence $5~\mathrm{s}$, plus height-resolved winds inferred from temporal tracking. The approach is trained in two stages on synthetic LES data and real-world camera data, using a sparse 3D transformer for refinement and CoTracker3 for wind retrieval. In a two-month deployment with six cameras over a $5~\mathrm{km} \times 5~\mathrm{km}$ area, Cloud4D achieves an order-of-magnitude improvement in space-time resolution relative to satellite products while maintaining $<10\%$ relative error against collocated radar and wind profiler measurements, enabling scalable, high-fidelity cloud observations. This work provides detailed, fast-changing cloud-property fields to support evaluation and development of high-resolution weather and climate models.

Abstract

There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four-dimensional cloud state using only synchronized ground-based cameras. Leveraging a homography-guided 2D-to-3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10\%$) against collocated radar measurements. Code and data are available on our project page https://cloud4d.jacob-lin.com/.

Figures (13)

• Figure 1: As a result of the cloud formation process, clouds are spatially structured into layers.
• Figure 2: Model overview. Our model estimates the liquid water content of clouds using a sparse set of ground-based cameras. a) Cloud Layer Model: We leverage an inductive bias on the spatial structure of clouds by defining a homography that maps images to cloud layers. The cloud homography is used to predict key 2.5D cloud properties, giving an initial estimate of the 3D cloud layer. b) 3D Refinement: A sparse transformer then refines the initial 3D field and estimates the final cloud liquid water content. c) Inference: Wind Retrieval: By tracking the motion of cloud reconstructions over time, our method retrieves a height and time-varying horizontal wind profile.
• Figure 3: Comparison with radar retrievals. To compare with radar values, we visualize our 3D liquid water content along a single ray over time. ERA5 captures coarse properties such as cloud heights and mean LWP. Comparatively, our method predicts high-resolution cloud properties that match up with radar retrievals.
• Figure 4: Qualitative comparison against satellite imagery. We render our 3D liquid water content predictions from a top-down orthographic view using Blender and compare against 2D image retrievals from Sentinel-2 and MODIS. Cloud4D estimates volumetric cloud properties every five seconds, while on average, Sentinel-2 takes one image every five days and MODIS once per day.
• Figure 5: Comparison of horizontal wind retrieval. Cloud4D is able to estimate height-varying horizontal wind vectors from the motion of our cloud predictions. Our wind profiles are of similar magnitude and direction to retrievals from a wind profiler. The arrow direction denotes the horizontal wind direction following the convention of a compass bearing.