Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

TL;DR

This work tackles the ill-posed problem of single-view reconstruction of volumetric density fields under complex light transport by introducing a diffusion-prior framework operating in a learned latent space. A novel monoplanar latent representation encodes 3D cloud densities efficiently, while a physically-based differentiable volume renderer provides gradients through light transport to steer posterior sampling toward plausible reconstructions $p(\theta|y;\phi)$. Key contributions include the Cloudy dataset of 1,000 synthetic clouds, the diffusion-friendly monoplanar decoder, and the Parametric Diffusion Posterior Sampling (PDPS) approach that integrates diffusion priors with differentiable rendering. Experiments show high-quality single-view cloud reconstructions, effective super-resolution, and the ability to recover lighting conditions, offering a practical tool for volumetric visualization and remote sensing under challenging illumination conditions.

Abstract

We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Figures (16)

• Figure 1: Given a single view ($y$) of a volume ($V$), we reconstruct a volume ($\hat{V}$) from its latent representation ($\theta$) that matches $y$ under the same lighting conditions, resulting in a synthesized view ($\hat{y}$). A differentiable volume renderer ($\mathcal{R}$) is used to optimize physical scene parameters ($\phi$) while simultaneously performing posterior sampling $p(\theta|y;\phi)$, conditioned on the observation, in the latent space of a trained diffusion model $p(\theta)$. Ambiguities due to the absence of information about unseen parts of the volume are reduced by gradually steering the reverse diffusion process toward the most plausible reconstruction under the given view (right section).
• Figure 2: Top images: Cloudy Dataset -- Photorealistic renderings of randomly selected clouds from our dataset, illustrating natural variations and details. Bottom images: Diffusion-based cloud synthesis -- Clouds generated with our diffusion model, demonstrating a convincing appearance under realistic lighting conditions and physical parameters.
• Figure 3: Implicit monoplanar representation.
• Figure 4: Diffusion Sampling. First column: A cloud from the Cloudy dataset. Subsequent columns show clouds generated by our diffusion model. First row shows the clouds under neutral lighting conditions, demonstrating realistic cloud-like formations. Bottom row shows cross-sectional slices through the volumes, demonstrating realistic interiors of diffused clouds.
• Figure 5: Diffusion Posterior Sampling. Given an observation and a differentiable process (differentiable volume rendering in our application), the denoising process is guided step-by-step toward matching the observation. From a different view, the reconstructed cloud may deviate from the ground truth, but the diffusion prior ensures that a realistic cloud is generated.