PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image

TL;DR

PhySIC addresses the challenge of reconstructing metrically accurate 3D humans and surrounding scenes from a single RGB image by jointly optimizing a metric-scale SMPL-X human mesh and a detailed scene geometry within a shared coordinate frame. The method uses a three-stage pipeline: Stage 1 builds a metric-scale scene by inpainting occluded regions and fusing metric depth with relative geometry; Stage 2 reconstructs and aligns the human mesh to the scene; Stage 3 performs a joint optimization with contact, interpenetration, and depth-consistency losses, augmented by occlusion handling, to produce physically plausible interactions. Across PROX and RICH, PhySIC achieves state-of-the-art pose and contact metrics while handling multiple humans and diverse environments, with end-to-end runtimes under 27 seconds per image. The work advances scalable single-image 3D human-centric scene understanding by integrating depth, geometry, and interaction priors into a unified optimization framework and releasing code for broad adoption.

Abstract

Reconstructing metrically accurate humans and their surrounding scenes from a single image is crucial for virtual reality, robotics, and comprehensive 3D scene understanding. However, existing methods struggle with depth ambiguity, occlusions, and physically inconsistent contacts. To address these challenges, we introduce PhySIC, a framework for physically plausible Human-Scene Interaction and Contact reconstruction. PhySIC recovers metrically consistent SMPL-X human meshes, dense scene surfaces, and vertex-level contact maps within a shared coordinate frame from a single RGB image. Starting from coarse monocular depth and body estimates, PhySIC performs occlusion-aware inpainting, fuses visible depth with unscaled geometry for a robust metric scaffold, and synthesizes missing support surfaces like floors. A confidence-weighted optimization refines body pose, camera parameters, and global scale by jointly enforcing depth alignment, contact priors, interpenetration avoidance, and 2D reprojection consistency. Explicit occlusion masking safeguards invisible regions against implausible configurations. PhySIC is efficient, requiring only 9 seconds for joint human-scene optimization and under 27 seconds end-to-end. It naturally handles multiple humans, enabling reconstruction of diverse interactions. Empirically, PhySIC outperforms single-image baselines, reducing mean per-vertex scene error from 641 mm to 227 mm, halving PA-MPJPE to 42 mm, and improving contact F1 from 0.09 to 0.51. Qualitative results show realistic foot-floor interactions, natural seating, and plausible reconstructions of heavily occluded furniture. By converting a single image into a physically plausible 3D human-scene pair, PhySIC advances scalable 3D scene understanding. Our implementation is publicly available at https://yuxuan-xue.com/physic.

Figures (9)

• Figure 1: Method overview. Given a single RGB image, we obtain accurate human, scene and contact reconstruction in 3D. We first obtain a complete metric scale scene with detailed geometry (Stage 1, \ref{['subsec:scene-init']}) and initialize human mesh which roughly aligns with the scene without contacts (Stage 2, \ref{['subsec:human-init']}). We then jointly optimize human and scene to satisfy contact constraints while avoiding penetrations (Stage 3, \ref{['subsec:joint-opt']}). https://unsplash.com/photos/woman-sitting-on-sofa-chair-beside-window-Gem-93Ea_RQ.
• Figure 2: Qualitative results on PROX dataset (row 1) and internet images (row 2-3). We compare the output of PhySIC with PROX hassan2019prox and HolisticMesh weng_holistic_2021. Note that we run PROX with our estimated scene on internet images as there is no scene scan available. Both PROX and HolisticMesh rely on predefined contact maps, hence are not robust to complex human poses and interactions. Our method reconstructs 3D scene and adapts contact optimization based on input, leading to more coherent reconstruction. Please refer Fig. \ref{['fig:qualitative_additional']} and Fig. \ref{['fig:qualitative_recons_2']} for more results.
• Figure 3: Qualitative results for contact estimation. We compare our approach against the state-of-the-art image-based contact predictor, DECO tripathi2023deco in both lab and wild setting. Note how our method improves the nuanced contact on arms and feet. Please refer Fig. \ref{['fig:qualitative_contact_2']} for further examples.
• Figure 4: Additional qualitative results for in-the-wild images. Please refer to Supp. Mat. for more results.
• Figure 5: Qualitative results for contact estimation. We compare our approach against the state-of-the-art image-based contact predictor, DECO.