PhyRecon: Physically Plausible Neural Scene Reconstruction

TL;DR

PHYRECON is introduced, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations, and exhibits superior physical stability in physical simulators, paving the way for future physics-based applications.

Abstract

We address the issue of physical implausibility in multi-view neural reconstruction. While implicit representations have gained popularity in multi-view 3D reconstruction, previous work struggles to yield physically plausible results, limiting their utility in domains requiring rigorous physical accuracy. This lack of plausibility stems from the absence of physics modeling in existing methods and their inability to recover intricate geometrical structures. In this paper, we introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations. PHYRECON features a novel differentiable particle-based physical simulator built on neural implicit representations. Central to this design is an efficient transformation between SDF-based implicit representations and explicit surface points via our proposed Surface Points Marching Cubes (SP-MC), enabling differentiable learning with both rendering and physical losses. Additionally, PHYRECON models both rendering and physical uncertainty to identify and compensate for inconsistent and inaccurate monocular geometric priors. The physical uncertainty further facilitates physics-guided pixel sampling to enhance the learning of slender structures. By integrating these techniques, our model supports differentiable joint modeling of appearance, geometry, and physics. Extensive experiments demonstrate that PHYRECON significantly improves the reconstruction quality. Our results also exhibit superior physical stability in physical simulators, with at least a 40% improvement across all datasets, paving the way for future physics-based applications.

Figures (10)

• Figure 1: Illustration of PhyRecon. We leverage both differentiable physics simulation and differentiable rendering to learn implicit surface representation. Results from previous methods li2023rico fail to remain stable in physical simulators or recover intricate geometries, while PhyRecon achieves significant improvements in both reconstruction quality and physical plausibility.
• Figure 2: Overview of PhyRecon. We incorporate explicit physical constraints in the neural scene reconstruction through a differentiable particle-based physical simulator and a differentiable transformation (i.e., spmc) between implicit surfaces and explicit surface points in \ref{['sec:method_simulator']}. To learn intricate 3D structures, we introduce rendering and physical uncertainty in \ref{['sec:method_uncertainty']} to address the inconsistencies in the geometric priors and guide the pixel sampling.
• Figure 3: Illustration of spmc. (a-b) We first shift the sdf grids, and (c) localize the zero-crossing vertices ${\bm{V}}$ (blue). (d) The coarse surface points ${\bm{P}}_\text{coarse}$ (black) are derived through linear interpolation and (e) the fine-grained points ${\bm{P}}_\text{fine}$ (purple) are obtained by querying the sdf network $f(\cdot)$.
• Figure 4: Qualitative results of indoor scene reconstruction. Examples from ScanNet++ yeshwanthliu2023scannetpp, ScanNet dai2017scannet, and Replica replica19arxiv demonstrate that our model produces higher quality reconstructions compared to the baselines. Our results contain finer details for slender structures (e.g., chair legs and objects on the table) and plausible support relations, which are highlighted in the zoom-in boxes.
• Figure 5: Object trajectory during simulation. Our method enhances the physical plausibility of the reconstruction results, which can remain stable during dropping simulation in Isaac Gym.