DNN-based 3D Cloud Retrieval for Variable Solar Illumination and Multiview Spaceborne Imaging

TL;DR

This work introduces the first scalable DNN-based system for 3-D cloud retrieval that accommodates varying camera positions and solar directions, and shows substantial improvements over previous state-of-the-art methods, particularly in handling variations in the sun’s zenith angle.

Abstract

Climate studies often rely on remotely sensed images to retrieve two-dimensional maps of cloud properties. To advance volumetric analysis, we focus on recovering the three-dimensional (3D) heterogeneous extinction coefficient field of shallow clouds using multiview remote sensing data. Climate research requires large-scale worldwide statistics. To enable scalable data processing, previous deep neural networks (DNNs) can infer at spaceborne remote sensing downlink rates. However, prior methods are limited to a fixed solar illumination direction. In this work, we introduce the first scalable DNN-based system for 3D cloud retrieval that accommodates varying camera poses and solar directions. By integrating multiview cloud intensity images with camera poses and solar direction data, we achieve greater flexibility in recovery. Training of the DNN is performed by a novel two-stage scheme to address the high number of degrees of freedom in this problem. Our approach shows substantial improvements over previous state-of-the-art, particularly in handling variations in the sun's zenith angle.

Figures (4)

• Figure 1: A) A satellite formation in orbit abserves a cloud. The uncontrolled scene is illuminated by the sun in direction ${\boldsymbol{\omega}}^{\rm sun}=[\omega^{\rm zenith}, \omega^{\rm azimuth}]$. Radiation that reaches a point ${\bf X}$ in the atmospheric domain is multiply scattered until sensed by camera sensor at ${\bf X}_c$. B) Cloud single-channel intensity images are shown from the same satellite position. The images change significantly when illuminated from a different direction.
• Figure 2: Architecture of PIVOT-CT. Feature extraction is performed on all images, resulting in a feature vector ${\bf v}({\bf X})$ for each atmospheric location ${\bf X}$, corresponding to the image pixels that are the geometric projections of ${\bf X}$. The atmospheric location ${\bf X}$ and camera positions $\{{\bf X}{c}\}_{c=1}^{N^{\rm cam}}$ are encoded into vectors ${\bf g}^{\rm domain}({\bf X})$ and $\{{\bf g}^{\rm cam}({\bf X}{c})\}_{c=1}^{N^{\rm cam}}$, respectively. Additionally, the illumination direction $\boldsymbol{\omega}^{\rm sun}$ is provided to the system. A zoom-in of the Sun Encoder module is presented. The sun direction $\boldsymbol{\omega}^{\rm sun}\in {\mathbb R}^2$ is embedded to a vector ${\bf g}^{\rm sun}$ by five fully connected layers, each followed by a ReLU activation. These feature vectors are then input into a decoder, which predicts the extinction coefficient $\hat{\beta}({\bf X})$.
• Figure 3: [Top] Visualizations of a test cloud's 3D extinction coefficient field and its recovery by VIP-CT and PIVOT-CT (ours). We show the relative error $\epsilon$ for each method. [Bottom] An illustration of the solar angle that VIP-CT was trained on and the true test scene solar angle. Scatter plots of the estimated $\hat{\beta}$ compared to the true $\beta^{\rm true}$ across all voxels, the red dashed line represents optimal recovery.
• Figure 4: Comparison of the PIVOT-CT (ours) and VIP-CT systems. This figure shows the $\epsilon$ measure (Eq. \ref{['eq:erros']}) as a function of the distance between the zenith angle of the test scenes and the zenith angle upon which VIP-CT was trained, $\omega^{\rm zenith}_{\rm fixed}$. The shaded area represents the standard deviation of $\epsilon$. The largest zenith angle difference of $50^\circ$ stands for the sun at the horizon.