PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF

TL;DR

PIE-NeRF presents a meshless physics-based framework that integrates nonlinear elastodynamics with NeRF to produce physically grounded, interactive deformations of NeRF scenes. By employing Q-GMLS with adaptive Augmented Poisson Disk sampling and per-IP energy integration, the method achieves robust, real-time simulations for hyperelastic materials and renders results with quadratic warping for faithful texture mapping. The approach demonstrates close agreement with ground-truth FEM and outperforms PAC-NeRF in rendering quality and speed, while accommodating topology changes and codimensional geometries. This work broadens NeRF's applicability to dynamic, physics-informed animations and offers a practical, scalable pathway for real-time, physics-accurate novel view synthesis.

Abstract

We show that physics-based simulations can be seamlessly integrated with NeRF to generate high-quality elastodynamics of real-world objects. Unlike existing methods, we discretize nonlinear hyperelasticity in a meshless way, obviating the necessity for intermediate auxiliary shape proxies like a tetrahedral mesh or voxel grid. A quadratic generalized moving least square (Q-GMLS) is employed to capture nonlinear dynamics and large deformation on the implicit model. Such meshless integration enables versatile simulations of complex and codimensional shapes. We adaptively place the least-square kernels according to the NeRF density field to significantly reduce the complexity of the nonlinear simulation. As a result, physically realistic animations can be conveniently synthesized using our method for a wide range of hyperelastic materials at an interactive rate. For more information, please visit our project page at https://fytalon.github.io/pienerf/.

Figures (12)

• Figure 1: Swaying plant. PIE-NeRF is an efficient and versatile pipeline that synthesizes physics-based novel motions of complex NeRF models interactively. In this example, the user interactively manipulates the plant by applying external forces with the mouse. The geometry of the plant is sampled in a meshless way, and a spatial model reduction is followed. We use 78 Q-GMLS kernels to capture the nonlinear dynamics of the plant in real-time. PIE-NeRF generates novel poses of the model from novel views in a physics-grounded way.
• Figure 2: Pipeline overview. The input of PIE-NeRF is the same as other NeRF-based frameworks, which consists of a collection of images of a static scene. An adaptive Poisson disk sampling is followed to query the 3D geometry of the model, which are sparsified into $n$ Q-GMLS kernels. Integrator points are placed over the model, including centers of Q-GMLS kernels (i.e., kernel IPs). Discretization at kernels and numerical integration at IPs enable efficient synthesis of novel and physics-based elastodynamic motions. The quadratic warping scheme helps to better retrieve the color/texture of a deformed spatial position to render the final result.
• Figure 3: Particle sampling. Our method is compatible with most sampling algorithms -- as long as particles cover the shape of the 3D model sufficiently well. In our implementation, we design a novel augmented Poisson disk sampling scheme that is fast and well captures the boundary of the model by default.
• Figure 4: Elastically deforming excavator. The excavator is a standard benchmark for NeRF-based frameworks. We use this classic model to showcase the capability of PIE-NeRF, which generates interesting and novel dynamic effects in real time.
• Figure 5: Interactive NeRF deformation. We developed an intuitive UI for users to interact with NeRF scenes like applying external forces and position constraints. Q-GMLS kernels can also be set adaptively to capture local dynamics as highlighted.