DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification

TL;DR

DecoDINO advances dense vertex-level 3D human–scene contact prediction by extending the DECO framework with two LoRA-tuned DINOv2 encoders (scene and body-part), patch-level cross-attention, and a semantic labeling head. It addresses DECO’s failure modes—class imbalance, occlusions, and soft surfaces—through a positive class balance loss and fine-grained patch interactions, achieving a notable improvement in binary contact F1 and geodesic error on the DAMON benchmark. A semantic per-vertex labeling capability is introduced, enabling object-level context for contacts, though semantic precision remains challenging. An attempted integration with a vision-language model did not improve performance, guiding future work toward denser labels, model distillation for efficiency, and broader benchmarking beyond DAMON.

Abstract

Accurate vertex-level contact prediction between humans and surrounding objects is a prerequisite for high fidelity human object interaction models used in robotics, AR/VR, and behavioral simulation. DECO was the first in the wild estimator for this task but is limited to binary contact maps and struggles with soft surfaces, occlusions, children, and false-positive foot contacts. We address these issues and introduce DecoDINO, a three-branch network based on DECO's framework. It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning. Vertex features are finally passed through a lightweight MLP with a softmax to assign semantic contact labels. We also tested a vision-language model (VLM) to integrate text features, but the simpler architecture performed better and was used instead. On the DAMON benchmark, DecoDINO (i) raises the binary-contact F1 score by 7$\%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels. Ablation studies show that LoRA fine-tuning and the dual encoders are key to these improvements. DecoDINO outperformed the challenge baseline in both tasks of the DAMON Challenge. Our code is available at https://github.com/DavidePasero/deco/tree/main.

Figures (14)

• Figure 1: DecoDINO improves DECO's performance on infering better dense vertex-level 3D contacts on the full human body. Given an RGB image, DecoDINO captures better binarry contact, handles failure cases (e.g. occlusion) and class imbalance (e.g. false foot contact prediction) better. Additionally, it enhances DECO with semantic classification, allowing DecoDINO to predict that the contact object is a couch.
• Figure 2: Number of images in the DAMON dataset with contact per body part. A body part is counted if any of its vertices are in contact.
• Figure 3: DECO's binary contact prediction on challenging scenario's
• Figure 4: Predicted scene and part segmentation with their ground truth (GT). Note: Predicted and ground-truth part masks use different label colors but refer to the same body parts (e.g., the head appears orange in the ground truth and pink in the prediction).
• Figure 5: Scene without a person. Predicted scene and part segmentation with the contact prediction on a SMPL body mesh on a scene without a person.