HOIGS: Human-Object Interaction Gaussian Splatting

Abstract

Reconstructing dynamic scenes with complex human-object interactions is a fundamental challenge in computer vision and graphics. Existing Gaussian Splatting methods either rely on human pose priors while neglecting dynamic objects, or approximate all motions within a single field, limiting their ability to capture interaction-rich dynamics. To address this gap, we propose Human-Object Interaction Gaussian Splatting (HOIGS), which explicitly models interaction-induced deformation between humans and objects through a cross-attention-based HOI module. Distinct deformation baselines are employed to extract features: HexPlane for humans and Cubic Hermite Spline (CHS) for objects. By integrating these heterogeneous features, HOIGS effectively captures interdependent motions and improves deformation estimation in scenarios involving occlusion, contact, and object manipulation. Comprehensive experiments on multiple datasets demonstrate that our method consistently outperforms state-of-the-art human-centric and 4D Gaussian approaches, highlighting the importance of explicitly modeling human-object interactions for high-fidelity reconstruction.

Figures (8)

• Figure 1: 4DGS-based and human-centric models fail to render HOI scenes.
• Figure 2: Overview of the Proposed Framework. Given an input video sequence, we first extract object-specific information to reconstruct the 3D object shape via a diffusion prior. Based on the reconstructed shape, we initialize 3D Gaussians for each keyframe and use a spline-based deformation as our baseline. Here, time-invariant and time varying HexPlane features are employed for canonical human and interaction modeling, respectively. The final deformation is driven by the HOI module, which integrates HexPlane-derived human features with CHS-based object features to model human-object interactions.
• Figure 3: Spline-based Object Motion Modeling. To ensure temporally continuous and physically plausible object trajectories, we parameterize the sequence of 3D Gaussians using a Cubic Hermite Spline (CHS).
• Figure 4: Cross-Attention-based HOI Module. The proposed architecture estimates human-object interactions using human body part features and per-Gaussian object representations.
• Figure 5: Qualitative comparison of reconstructed rendered views. We compare our method (HOIGS) against state-of-the-art baselines on both the HOSNeRFliu2023hosnerf and BEHAVEbhatnagar2022behave datasets. We display the full-frame (top) rendering and a zoom-in (bottom) of the red Region of Interest (ROI).