Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data

TL;DR

The paper tackles reconstructing dynamic fluid flows from sparse multi-view RGB videos without explicit lighting or geometry inputs. It introduces a time-varying neural radiance field $F_{vis}$ and a velocity field $F_{hid}$, trained end-to-end with image supervision and physics priors derived from Navier–Stokes equations, augmented by a pre-trained density-to-velocity model. A layer-by-layer progressive growing strategy and a ghost-density regularization mitigate color-density ambiguity, while a hybrid static-dynamic extension enables obstacle interaction without manual labeling. The approach achieves high-quality density, velocity, and vorticity reconstructions on synthetic and real data, including scenes with arbitrary static obstacles and complex lighting, and demonstrates robust performance under relaxed inputs. This work advances in-the-wild fluid capture by integrating differentiable rendering, PINN-based dynamics, and model-based priors into a unified neural representation for fluids.

Abstract

High-fidelity reconstruction of fluids from sparse multiview RGB videos remains a formidable challenge due to the complexity of the underlying physics as well as complex occlusion and lighting in captures. Existing solutions either assume knowledge of obstacles and lighting, or only focus on simple fluid scenes without obstacles or complex lighting, and thus are unsuitable for real-world scenes with unknown lighting or arbitrary obstacles. We present the first method to reconstruct dynamic fluid by leveraging the governing physics (ie, Navier -Stokes equations) in an end-to-end optimization from sparse videos without taking lighting conditions, geometry information, or boundary conditions as input. We provide a continuous spatio-temporal scene representation using neural networks as the ansatz of density and velocity solution functions for fluids as well as the radiance field for static objects. With a hybrid architecture that separates static and dynamic contents, fluid interactions with static obstacles are reconstructed for the first time without additional geometry input or human labeling. By augmenting time-varying neural radiance fields with physics-informed deep learning, our method benefits from the supervision of images and physical priors. To achieve robust optimization from sparse views, we introduced a layer-by-layer growing strategy to progressively increase the network capacity. Using progressively growing models with a new regularization term, we manage to disentangle density-color ambiguity in radiance fields without overfitting. A pretrained density-to-velocity fluid model is leveraged in addition as the data prior to avoid suboptimal velocity which underestimates vorticity but trivially fulfills physical equations. Our method exhibits high-quality results with relaxed constraints and strong flexibility on a representative set of synthetic and real flow captures.

Figures (13)

• Figure 1: Modules and supervision in our method. With the comprehensive supervision of images, physical-priors, and a published fluid network, we present neural fluid fields with radiance fields for static and dynamic components, and velocity fields for fluids.
• Figure 2: The VGG loss helps improve the perceptual quality of the reconstructed results and capture more high frequency details.
• Figure 3: The "ghost density" artifact. a) Density profile (side and top views) during training. b),c), and d) Rendering results using a dark green background.
• Figure 4: We use hybrid Models to learn radiance fields for static obstacles and dynamic fluids. The velocity model is only related to the density of fluids.
• Figure 5: Comparisons of rendering results on the ScalarFlow Dataset with synthetic (1st row) and real (2nd row) cases.