DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction

TL;DR

DecoupledGaussian addresses the challenge of decoupling static objects from their contact surfaces in real-world videos to enable physics-based interaction. It advances a geometry-centric restoration pipeline that combines planar-based Gaussian Splatting, joint Poisson fields, proxy points, and UNCE to recover accurate object and scene geometry before MLS-MPM-based simulation. The method demonstrates superior restoration quality and interactive realism across diverse real-world datasets, outperforming baselines in both static restoration metrics and dynamic simulations, with efficient, real-time capable performance. By enabling realistic detachment, collision, and fracture under user impulses, this approach holds practical impact for VR, robotics, and autonomous driving applications, while outlining clear pathways for handling more complex scenes and texture-generation challenges.

Abstract

We present DecoupledGaussian, a novel system that decouples static objects from their contacted surfaces captured in-the-wild videos, a key prerequisite for realistic Newtonian-based physical simulations. Unlike prior methods focused on synthetic data or elastic jittering along the contact surface, which prevent objects from fully detaching or moving independently, DecoupledGaussian allows for significant positional changes without being constrained by the initial contacted surface. Recognizing the limitations of current 2D inpainting tools for restoring 3D locations, our approach proposes joint Poisson fields to repair and expand the Gaussians of both objects and contacted scenes after separation. This is complemented by a multi-carve strategy to refine the object's geometry. Our system enables realistic simulations of decoupling motions, collisions, and fractures driven by user-specified impulses, supporting complex interactions within and across multiple scenes. We validate DecoupledGaussian through a comprehensive user study and quantitative benchmarks. This system enhances digital interaction with objects and scenes in real-world environments, benefiting industries such as VR, robotics, and autonomous driving. Our project page is at: https://wangmiaowei.github.io/DecoupledGaussian.github.io/.

Figures (11)

• Figure 1: DecoupledGaussian decomposes static objects and contacted scenes from videos or multi-view images, enabling simulations like scene collisions (Top) and object melting with material adjustments (Bottom). See the supplementary video for the full sequences.
• Figure 2: Inpainting tools (LaMa suvorov2022resolution; PhotoRoom photoroom_remove_object) introduce artifacts and inconsistent textures across frames.
• Figure 3: System Overview. DecoupledGaussian is an interactive simulation system that enables objects to detach from their initial contact surfaces after applying our proposed restoration pipeline, driven by user-specified impulses (red arrow on the right).
• Figure 4: Joint Poisson Fields $\mathcal{W}$ first reconstruct $\mathcal{O}$ and $\mathcal{S}$ independently, then resolve conflicts (red area) by defining a boundary that separates them into distinct, non-intersecting entities.
• Figure 5: Ablation for $\mathcal{P}_\mathcal{O}$. Independent Poisson reconstruction of object $\mathcal{O}$ using Gaussian centers $\{ \boldsymbol{k}_g \}_{g \in \mathcal{O}}$ yields poor mesh quality compared to using proxy points $\mathcal{P}_\mathcal{O}$. Our joint Poisson field $\mathcal{W}$, which integrates the scene surface $\mathcal{S}$, effectively removes the overextended regions (highlighted in red). The final dense points $P_\mathcal{O}$ are then combined with proxy points $\mathcal{P}_\mathcal{O}$ for Gaussian restoration and continuum simulation.