Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimization

TL;DR

The paper addresses geometry-agnostic system identification under sparse-view conditions by integrating Lagrangian particle optimization (LPO) with physics-informed PAC-NeRF. LPO shifts geometry optimization from Eulerian grids to Lagrangian space, enforcing MPM-based physical constraints so geometry can be corrected across the entire video sequence. It also introduces iterative geometry–physics optimization to reuse geometry corrections for reidentifying material properties, improving both geometry and physical parameters in challenging few-shot scenarios. Experiments on a nine-scene continuum-material dataset show that LPO consistently enhances geometric correction and physical property identification, and the approach generalizes beyond PAC-NeRF to other physics-informed models. The work thus broadens the applicability of differentiable physics-augmented neural radiance fields to sparse-view contexts while maintaining interpretability through explicit continuum mechanics constraints.

Abstract

Geometry-agnostic system identification is a technique for identifying the geometry and physical properties of an object from video sequences without any geometric assumptions. Recently, physics-augmented continuum neural radiance fields (PAC-NeRF) has demonstrated promising results for this technique by utilizing a hybrid Eulerian-Lagrangian representation, in which the geometry is represented by the Eulerian grid representations of NeRF, the physics is described by a material point method (MPM), and they are connected via Lagrangian particles. However, a notable limitation of PAC-NeRF is that its performance is sensitive to the learning of the geometry from the first frames owing to its two-step optimization. First, the grid representations are optimized with the first frames of video sequences, and then the physical properties are optimized through video sequences utilizing the fixed first-frame grid representations. This limitation can be critical when learning of the geometric structure is difficult, for example, in a few-shot (sparse view) setting. To overcome this limitation, we propose Lagrangian particle optimization (LPO), in which the positions and features of particles are optimized through video sequences in Lagrangian space. This method allows for the optimization of the geometric structure across the entire video sequence within the physical constraints imposed by the MPM. The experimental results demonstrate that the LPO is useful for geometric correction and physical identification in sparse-view settings.

Figures (9)

• Figure 1: Impact of the proposed Lagrangian particle optimization (LPO) in sparse-view geometric-agnostic system identification. We aim to identify the geometry and physical properties of an object from visual observations without any geometric assumptions in severe (e.g., sparse-view) settings. As shown in (b), a standard PAC-NeRF XLiICLR2023 has difficulty learning the geometry in a sparse-view setting (particularly, when the number of views is three). Consequently, it also fails to estimate the physical properties (Young’s modulus $E$ and Poisson's ratio $\nu$). As shown in (c), LPO is useful for correcting this failure and succeeds in improving the identification of both geometry and physical properties.
• Figure 2: Optimization pipelines of PAC-NeRF (1)(2) and the proposed LPO (3).
• Figure 3: Qualitative comparisons among PAC-NeRF-3v/3v$^\dag$, +LPO, and +LPO$^4$ on Newtonian fluids (Droplet and Letter). Blue fonts indicate the scores obtained by the baselines (PAC-NeRF-3v/3v$^\dag$). Red fonts indicate the scores obtained by the proposed methods (+LPO and +LPO$^4$). Given the initial estimation by the baseline (b)(e), +LPO first corrects the geometric structures (including appearance and shape) (c)(f). By repeatedly conducting physical identification and geometric correction via Algorithm \ref{['alg:iterative_optimization']}, +LPO$^4$ reidentifies physical properties and recorrects geometric structures (d)(g). In the Droplet scene, the bottom of the droplet is sharply pointed, and its tip is whitened in the baseline (b)(e). They are gradually mitigated by applying +LPO (c)(f) and +LPO$^4$ (d)(g). In the Letter scene, +LPO (c)(f) and +LPO$^4$ (d)(g) succeed in gradually eliminating artifacts existing in the vicinity of the left line of the letter "R.", which arise in the baselines (b)(e).
• Figure 4: Qualitative comparisons among PAC-NeRF-3v/3v$^\dag$, +LPO, and +LPO$^4$ on non-Newtonian fluids (Cream and Toothpaste). Blue fonts indicate the scores obtained by the baselines (PAC-NeRF-3v/3v$^\dag$). Red fonts indicate the scores obtained by the proposed methods (+LPO and +LPO$^4$). Given the initial estimation by the baseline (b)(e), +LPO first corrects the geometric structures (including appearance and shape) (c)(f). By repeatedly conducting physical identification and geometric correction via Algorithm \ref{['alg:iterative_optimization']}, +LPO$^4$ reidentifies physical properties and recorrects geometric structures (d)(g). In the Cream scene, the baselines (b)(e) fail to color the materials correctly. This failure is alleviated by +LPO (c)(f) and further mitigated by +LPO$^4$ (d)(g). In the Toothpaste scene, the baselines (b)(e) make the material darker color than that in the ground truth (a). +LPO (c)(f) makes the material brighter, and +LPO$^4$ (d)(g) obtains the color closer to the ground truth (a). This effect is pronounced when PAC-NeRF-3v$^\dag$ (e) is used as a baseline.
• Figure 5: Qualitative comparisons among PAC-NeRF-3v/3v$^\dag$, +LPO, and +LPO$^4$ on elastic materials (Torus and Bird). Blue fonts indicate the scores obtained by the baselines (PAC-NeRF-3v/3v$^\dag$). Red fonts indicate the scores obtained by the proposed methods (+LPO and +LPO$^4$). Given the initial estimation by the baseline (b)(e), +LPO first corrects the geometric structures (including appearance and shape) (c)(f). By repeatedly conducting physical identification and geometric correction via Algorithm \ref{['alg:iterative_optimization']}, +LPO$^4$ reidentifies physical properties and recorrects geometric structures (d)(g). In the Torus scene, the baselines (b)(e) have difficulty correctly capturing color and shape. They are improved by applying +LPO (c)(f), and the fine details are also improved by utilizing +LPO$^4$ (d)(g). Also, in the Bird scene, the baselines (b)(e) fail to capture color and shape correctly. The shape (e.g., the directions of the tail) is first corrected by +LPO (c)(f), and then the color is corrected by +LPO$^4$ (d)(g).