InteractVLM: 3D Interaction Reasoning from 2D Foundational Models

TL;DR

InteractVLM addresses 3D contact reasoning for human–object interaction from a single image by leveraging large vision–language models through a Render-Localize-Lift pipeline and a novel MV-Loc module that enforces multi-view consistency. It introduces Semantic Human Contact to condition body contacts on object semantics and demonstrates strong gains in binary and semantic contact estimation as well as object affordance prediction, all while reducing reliance on expensive 3D annotations. The method enables joint 3D reconstruction of humans and objects by constraining an optimization with inferred contacts, achieving realistic HOI from wild images. By exploiting unpaired data and textual cues via VLMs and a 3D-to-2D rendering bridge, InteractVLM scales across many categories and offers practical 3D HOI understanding with partial supervision.

Abstract

We introduce InteractVLM, a novel method to estimate 3D contact points on human bodies and objects from single in-the-wild images, enabling accurate human-object joint reconstruction in 3D. This is challenging due to occlusions, depth ambiguities, and widely varying object shapes. Existing methods rely on 3D contact annotations collected via expensive motion-capture systems or tedious manual labeling, limiting scalability and generalization. To overcome this, InteractVLM harnesses the broad visual knowledge of large Vision-Language Models (VLMs), fine-tuned with limited 3D contact data. However, directly applying these models is non-trivial, as they reason only in 2D, while human-object contact is inherently 3D. Thus we introduce a novel Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D space via multi-view rendering, (2) trains a novel multi-view localization model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D. Additionally, we propose a new task called Semantic Human Contact estimation, where human contact predictions are conditioned explicitly on object semantics, enabling richer interaction modeling. InteractVLM outperforms existing work on contact estimation and also facilitates 3D reconstruction from an in-the wild image. Code and models are available at https://interactvlm.is.tue.mpg.de.

Figures (11)

• Figure 1: We present InteractVLM, a novel method for estimating contact points on both human bodies and objects from a single in-the-wild image, shown here as red patches. Our method goes beyond traditional binary contact estimation methods by estimating contact points on a human in relation to a specified object. We do so by leveraging the broad visual knowledge of a large Visual Language Model.
• Figure 2: Overview of InteractVLM. Given a color image, our VLM performs the core reasoning, and guides a novel MV-Loc model to localize contacts on both bodies and objects in 3D. Here we show only the body; for details, and object contact, see \ref{['fig:method']}.
• Figure 3: Method overview. Given a single in-the-wild color image, our novel InteractVLM method estimates 3D contact points on both humans and objects (a). Then, we reconstruct a 3D human and object in interaction by exploiting these contacts (b). More specifically: (a) Contact estimation. Given an image, $I$, and prompt text, $T_{inp}$, our VLM, $\Psi$, produces contact tokens for humans and objects, < HCON> and < OCON>, which are projected ($\Gamma$) into feature embeddings, $E^{H}$ and $E^{O}$. These guide a "Multi-View [contact] Localization" model. This renders the 3D human and object geometry via cameras, ${K}$, into multi-view 2D renders and passes these to encoder, $\Theta$, while decoders, $\Omega^H$, $\Omega^O$, estimate and highlight 2D contacts in these renders. Then, the FeatLift module, $\Phi$, transforms the VLM's features ($E^{H}$, $E^{O}$) to become 3D-aware ($E^{H}_{3D}$, $E^{O}_{3D}$) by exploiting the camera parameters, ${K}$. A final module lifts the detected 2D contacts to 3D. (b) 3D HOI reconstruction. For joint human-object reconstruction, we use InteractVLM's inferred contacts in an optimization framework.
• Figure 4: "Semantic Human Contact" estimation (\ref{['sec:semantic_human_contact']}). Given an image and an object label, InteractVLM infers body contacts for this object. InteractVLM outperforms a Semantic-DECO tripathi2023deco baseline. Objects are shown in green circles, and contacts as red patches.
• Figure 5: InteractVLM's reliance on 3D annotations. We evaluate performance for "binary human contact" (F1 score, Y-axis) for models trained on a varying percentage of DAMON tripathi2023deco training data (X-axis). The DECO baseline trains on 100% of DAMON. Instead, InteractVLM trains on a varying (smaller) portion of this dataset. Yet, it achieves a significantly higher performance, by leveraging the broad visual knowledge of foundation models.