GRACE: Estimating Geometry-level 3D Human-Scene Contact from 2D Images

TL;DR

This work tackles the challenge of geometry-level 3D human–scene contact estimation from 2D images by moving beyond fixed SMPL-topology mappings. GRACE fuses 2D interaction semantics with 3D geometric priors through a hierarchical cross-modal framework that operates on unstructured point clouds, using a point-cloud encoder–decoder and HFEM/MFFM modules to predict dense per-vertex contact probabilities without relying on predefined mesh topology. Key contributions include a novel multi-level feature extraction and cross-modal fusion strategy, a geometry-aware decoding strategy, and an enhanced evaluation metric (geo.sum) that captures both false positives and negatives. The method achieves state-of-the-art results on HSI benchmarks and demonstrates robust generalization to non-SMPL and unstructured human shapes, signaling strong potential for real-world applications in AR/VR, robotics, and embodied AI.

Abstract

Estimating the geometry level of human-scene contact aims to ground specific contact surface points at 3D human geometries, which provides a spatial prior and bridges the interaction between human and scene, supporting applications such as human behavior analysis, embodied AI, and AR/VR. To complete the task, existing approaches predominantly rely on parametric human models (e.g., SMPL), which establish correspondences between images and contact regions through fixed SMPL vertex sequences. This actually completes the mapping from image features to an ordered sequence. However, this approach lacks consideration of geometry, limiting its generalizability in distinct human geometries. In this paper, we introduce GRACE (Geometry-level Reasoning for 3D Human-scene Contact Estimation), a new paradigm for 3D human contact estimation. GRACE incorporates a point cloud encoder-decoder architecture along with a hierarchical feature extraction and fusion module, enabling the effective integration of 3D human geometric structures with 2D interaction semantics derived from images. Guided by visual cues, GRACE establishes an implicit mapping from geometric features to the vertex space of the 3D human mesh, thereby achieving accurate modeling of contact regions. This design ensures high prediction accuracy and endows the framework with strong generalization capability across diverse human geometries. Extensive experiments on multiple benchmark datasets demonstrate that GRACE achieves state-of-the-art performance in contact estimation, with additional results further validating its robust generalization to unstructured human point clouds.

Figures (7)

• Figure 1: Given a monocular image and its paired arbitrary 3D human point cloud, GRACE accurately predicts human-scene contact. The yellow-highlighted regions indicate the predicted dense contact areas.
• Figure 2: Method. Overview of Geometry-level Reasoning for 3D Human-scene Contact Estimation Network(GRACE), it first extracts image features $\mathbf{F}_{i},\mathbf{F}_{p}$ and point cloud features $\mathbf{F}_{h}$, then uses Hierarchical Feature Extraction Module (Sec. \ref{['sec:3.2']}) to extract $\hat{\mathbf{F}}_p$, $\hat{\mathbf{F}}_i$, $\mathbf{F}_{hp}$, $\mathbf{F}_{hg}$ and $\mathbf{F}_{ig}$. Next, Multi-level Feature Fusion (Sec. \ref{['sec:3.3']}) aligns $\mathbf{\Theta}_{1/2}$ and $\mathbf{F}_{g}$ to obtain the 3D contact fusion feature $\hat{\mathbf{F}}_{c}$. Finally, $\hat{\mathbf{F}}_{c}$ is sent to the point decoder to obtain the final contact prediction result $\hat{y}$.
• Figure 3: Visualization Results. Qualitative evaluation of GRACE, DECO tripathi2023deco and BSTRO Huang:CVPR:2022 , alongside Ground Truth. The yellow regions indicate the areas of contact between the human body and the scene.
• Figure 4: Ablation of part feature branch. The result of contact estimation with ($\bm{w }$), without($\bm{w/o }$) part feature branch of image and point cloud.
• Figure 5: Ablation of global feature branch. The result of contact estimation with ($\bm{w }$), without($\bm{w/o }$) global feature branch of image and point cloud.