Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians

TL;DR

Spring-Gaus introduces a simulatable elastic-object representation by coupling a differentiable 3D Gaussian appearance/geometry model with a learnable 3D Spring-Mass dynamics. The approach decouples appearance/geometry optimization from physics parameter learning, enabling efficient inverse parameter estimation from multi-view videos and differentiable rendering for forward simulation. Empirical results on synthetic and real data demonstrate accurate reconstruction and short-horizon future prediction, with the ability to generalize to new boundary and environmental conditions. This work provides a practical pathway to creating reusable digital assets of heterogeneous elastic objects for predictive perception and interactive robotics.

Abstract

Reconstructing and simulating elastic objects from visual observations is crucial for applications in computer vision and robotics. Existing methods, such as 3D Gaussians, model 3D appearance and geometry, but lack the ability to estimate physical properties for objects and simulate them. The core challenge lies in integrating an expressive yet efficient physical dynamics model. We propose Spring-Gaus, a 3D physical object representation for reconstructing and simulating elastic objects from videos of the object from multiple viewpoints. In particular, we develop and integrate a 3D Spring-Mass model into 3D Gaussian kernels, enabling the reconstruction of the visual appearance, shape, and physical dynamics of the object. Our approach enables future prediction and simulation under various initial states and environmental properties. We evaluate Spring-Gaus on both synthetic and real-world datasets, demonstrating accurate reconstruction and simulation of elastic objects. Project page: https://zlicheng.com/spring_gaus/.

Figures (9)

• Figure 1: Spring-Gaus reconstructs the appearance, geometry, and physical dynamic properties of elastic objects from video observations. Spring-Gaus enables future predictions and simulations under different initial states and environmental conditions.
• Figure 2: Overview of Spring-Gaus reconstruction pipeline:(a) Static Scene Reconstruction: We start by reconstructing static 3D Gaussians from the first frames of the multiview videos. (b) Refining 3D Gaussians: We extract a set of anchor points to allow efficient simulation, which leads to appearance drift. We refine the 3D Gaussians to better model the appearance during simulation. (c) Dynamic Reconstruction: Our 3D Spring-Mass model simulates anchor points and updates the positions of Gaussian kernels. Upon completion of optimization, we obtain a simulatable 3D object that accurately models its dynamics.
• Figure 3: Registration from static scene to dynamic scene for real-world sample.
• Figure 4: Qualitative results on synthetic data. Compared with PAC-NeRF li2023pacnerf, Dynamic 3D Gaussians luiten2023dynamic and 4D Gaussian Splatting wu20234dgaussians, Spring-Gaus can maintain a good geometry and appearance while reconstructing reasonable dynamics.
• Figure 5: Qualitative results of future prediction on real-world samples. Predicted dynamics closely follow real observations.