Open-Vocabulary Functional 3D Human-Scene Interaction Generation

TL;DR

FunHSI tackles the problem of generating functionally correct 3D human–scene interactions in unseen environments by coupling functionality-grounding with a contact-graph representation and a two-stage, optimization-based body refinement. The method uses open-vocabulary task prompts and foundation models to identify functional elements, reason about contact patterns via an LLM-generated graph, and then synthesize and refine a 3D SMPL-X human that plausibly touches targeted scene elements. Key contributions include a functionality-aware contact reasoning module, a generator–evaluator inpainting loop for robust human initialization, contact-graph refinement to resolve laterality, and a two-stage body refinement with collision/contact losses. The approach demonstrates strong functional grounding, generalization to city scenes, and favorable user perception, offering a practical pathway toward semantically grounded embodied interaction synthesis in real-world environments.

Abstract

Generating 3D humans that functionally interact with 3D scenes remains an open problem with applications in embodied AI, robotics, and interactive content creation. The key challenge involves reasoning about both the semantics of functional elements in 3D scenes and the 3D human poses required to achieve functionality-aware interaction. Unfortunately, existing methods typically lack explicit reasoning over object functionality and the corresponding human-scene contact, resulting in implausible or functionally incorrect interactions. In this work, we propose FunHSI, a training-free, functionality-driven framework that enables functionally correct human-scene interactions from open-vocabulary task prompts. Given a task prompt, FunHSI performs functionality-aware contact reasoning to identify functional scene elements, reconstruct their 3D geometry, and model high-level interactions via a contact graph. We then leverage vision-language models to synthesize a human performing the task in the image and estimate proposed 3D body and hand poses. Finally, the proposed 3D body configuration is refined via stage-wise optimization to ensure physical plausibility and functional correctness. In contrast to existing methods, FunHSI not only synthesizes more plausible general 3D interactions, such as "sitting on a sofa'', while supporting fine-grained functional human-scene interactions, e.g., "increasing the room temperature''. Extensive experiments demonstrate that FunHSI consistently generates functionally correct and physically plausible human-scene interactions across diverse indoor and outdoor scenes.

Figures (15)

• Figure 1: Given a set of RGB-D images, their camera parameters, and a task description, our method automatically generates a 3D human body that interacts with the appropriate functional element in the scene. Leveraging the generalization power of foundation models, our method is training-free and applicable to unseen environments and open-vocabulary task descriptions in a zero-shot manner.
• Figure 2: Illustration of our FunHSI method. Given a set of posed RGB-D images, and a task prompt, FunHSI generates 3D humans interacting with functional elements (e.g., "knob" or "switch") to perform the specified task. First, functionality-aware contact reasoning detects elements to be interacted with, constructs a contact graph, and performs segmentation. Next, functionality-aware body initialization performs human inpainting, pose estimation, and contact graph refinement, where a generator–evaluator loop ensures no hallucination and correct contact targeting. Finally, body refinement performs optimization to improve the body configuration and the contact.
• Figure 3: Visualization of the human inpainting optimization process. By automatically evaluating the human inpainting results, the image generation process is optimized to produce more reliable outcomes, thus strongly facilitating the subsequent body optimization step.
• Figure 4: Qualitative results on SceneFun3D for general human-scene interaction. We compare GenZI*, GenHSI*, and our FunHSI with non-functional prompts such as sitting, squatting, and walking.
• Figure 5: Qualitative results on SceneFun3D for functional human-scene interaction. Given open-vocabulary functional commands (e.g., adjusting temperature, dialing a number, switching a radio station) and posed RGB-D inputs, we compare GenZI*, GenHSI*, and our FunHSI. Existing methods struggle to reason about task intent and often interact with incorrect objects or miss fine-grained functional components. In contrast, FunHSI accurately identifies task-relevant functional elements and generates physically plausible 3D human poses that establish correct contacts with both large objects and small functional parts (e.g., knobs, dials, cabinet handles), demonstrating robust functional grounding and contact reasoning.