Learned 3D volumetric recovery of clouds and its uncertainty for climate analysis

Abstract

Significant uncertainty in climate prediction and cloud physics is tied to observational gaps relating to shallow scattered clouds. Addressing these challenges requires remote sensing of their three-dimensional (3D) heterogeneous volumetric scattering content. This calls for passive scattering computed tomography (CT). We design a learning-based model (ProbCT) to achieve CT of such clouds, based on noisy multi-view spaceborne images. ProbCT infers - for the first time - the posterior probability distribution of the heterogeneous extinction coefficient, per 3D location. This yields arbitrary valuable statistics, e.g., the 3D field of the most probable extinction and its uncertainty. ProbCT uses a neural-field representation, making essentially real-time inference. ProbCT undergoes supervised training by a new labeled multi-class database of physics-based volumetric fields of clouds and their corresponding images. To improve out-of-distribution inference, we incorporate self-supervised learning through differential rendering. We demonstrate the approach in simulations and on real-world data, and indicate the relevance of 3D recovery and uncertainty to precipitation and renewable energy.

Figures (9)

• Figure 1: [A] Images are captured by a coordinated satellite formation (as CloudCT). The images are processed by ProbCT. ProbCT infers the posterior probability distribution of the cloud extinction coefficient at any point in a 3D domain. It thus infers a volumetric map of probability distributions (a distribution per location). This facilitates several product, such as: 3D maps of the most probable value and uncertainty of the cloud extinction coefficient, precipitation forecast or uncertainty of solar power at ground level. [B] Several classes of clouds, imaged by the VIIRS instrument onboard the NOAA-20 satellite: {1} Anvils of continental deep convective systems; {2} Marine deep convective systems; Marine stratocumulus deck with closed {3} and open {4} cells; {5} Trade cumulus. [C] Statistics of four classes of simulated cumulus clouds, based on empirical environmental boundary conditions.
• Figure 2: [A] A physics-based cloud simulator generates a random 3D cloud fields ${\boldsymbol \beta}^{\rm true}$. The scene is then physically rendered to yield corresponding multi-view images ${\bf y}$. [B] Supervised volumetric training of ProbCT (tuning the DNN parameters, $\boldsymbol{\Theta}$). ProbCT trains on pairs of labeled data $({\boldsymbol \beta}^{\rm true},{\bf y})$ to estimate a posterior probability distribution, ${\hat{P}}_{{\bf X}}(\beta|{\bf y},{\boldsymbol \Theta})$, per 3D location ${\bf X}$. This probability is compared to the true probability at this location, ${ P}^{\rm true}_{{\bf X}}(\beta)$, which is a delta function. The comparison uses the Kullback-Leibler divergence. [C] Self-supervised training, to improve inference of OOD clouds. Given observed cloud images, ProbCT estimates ${\hat{P}}_{{\bf X}}(\beta|{\bf y},{\boldsymbol \Theta})$, yielding a maximum a-posteriori estimate ${\hat{\boldsymbol \beta}}$. Using ${\hat{\boldsymbol \beta}}$ in RT yields re-projected rendered images. These images are compared to the observed images, to update ${\boldsymbol \Theta}$.
• Figure 3: [A] Visualizations of 3D volumetric fields by MIP at $45^\circ$ off-nadir: A labeled BOMEX50 test cloud, its estimation error and uncertainty (normalized entropy), which increases at the cloud core. [B] Sample inferred probability distributions from a cloud shown in [A] (an OOD test), normalized by the MAP value. [C] Inferred results at 2000 voxels, randomly sampled across the BOMEX50 test set. A high inferred normalized entropy (uncertainty) indeed implies a possible large absolute error. Large errors of $\hat{\beta}$ rarely occur when the inferred entropy is low. [D] Learning the posterior probability distribution of clouds that differ by a single voxel (pointed out by a black arrow). The cloud is visualized by MIP at $45^\circ$ off-nadir. [Right] Blue: the prior probability distribution from which $\beta$ at the voxel is drawn. Green: the sharply peaked true posterior of $\beta$ in this voxel of a test cloud, by $N^{\rm cam}=10$. Other lines plot ProbCT inferences for different number of views $N^{\rm cam}$. [E1] A recovered BOMEX500 test scene. The uncertainty in ${\hat{\boldsymbol\beta}}$ propagates to downstream forecasting tasks: [E2] renewable solar power generation on the ground gristey2020surface (see Eq. \ref{['eq:rel_responce']}) and [E3] droplet effective radius $r^{\rm e}$. A value $r^{\rm e}>14\mu{\rm m}$ is a precipitation triggerrosenfeld1994retrieving, yielding rain and dramatic shortening of cloud life.
• Figure 4: Real-world experiment. [A] Nadir image from a NASA AirMSPI flight at 20:27GMT on February 6, 2010 over 32N 123W. Green rectangles: regions used for self-supervised training. Red rectangle: a test domain. [B] Comparing (visually and by a scatter plot) an AirMSPI image excluded from inference vs. an image rendered in the corresponding viewpoint, based on the inferred cloud. For clarity, the scatter-plot uses $1\%$ of the image pixels. [C] MIP of the inferred MAP ${\hat{\beta}}$ of the cloud, MIP of the uncertainty (normalized entropy), and MIP of an estimated adiabatic fraction (AF). [D] Histogram of the estimated AF. Bar colors represent voxel distance from the cloud center $\bf O$. As expected, the AF decreases as distance increases from the cloud core.
• Figure 5: [A] The ProbCT architecture. A 3D scene is observed from $N^{\rm cam}$ viewpoints, yielding multi-image data ${\bf y}$. All images are processed by the same feature pyramid network, to extract corresponding image feature maps. Per location ${\bf X}$ the feature maps are sampled at spatial coordinates $\{ {\bf x}_c\}_{c=1}^{N^{\rm cam}}$ corresponding to geometric projections of ${\bf X}$. This leads to a vector ${\bf v}({\bf X})$ of features from all images. 3D coordinates of the location ${\bf X}$ and locations $\{{\bf X}_{c}\}_{c=1}^{N^{\rm cam}}$ of the viewpoints are processed using coordinate encoders, resulting respectively in geometric feature vectors ${\bf g}^{\rm domain}({\bf X})$ and $\{{\bf g}^{\rm cam}({\bf X}_{c})\}_{c=1}^{N^{\rm cam}}$. These vectors are passed to a decoder that infers the posterior probability distribution of the extinction coefficient ${P}_{{\bf X}}(\beta|{\bf y},{\boldsymbol \Theta})$. [B] Visualization of the differential ${\tt Smoothmax}$ (Boltzmann) operator asadi2017alternative in \ref{['eq:amp_p']}. This form is used during self-supervised training. [C] At boundary point ${\bf X}_0$, known radiance $I({\bf X}_0, {\boldsymbol \omega}_0)$ is incident in direction ${\boldsymbol \omega}_0$. Radiance scatters multiple times in the domain. The 3D functions $J$ (Eq. \ref{['eq:J']}) and the extinction coefficient $\beta$ define the radiance field $I$ by \ref{['eq:I']}. Pixel $p$ of camera $c$ corresponds line of sight to direction ${\boldsymbol \omega}_{c,p}$. This pixel samples the radiance $I({\bf X}_{c}, {\boldsymbol \omega}_{c,p})$.