THOR: Text to Human-Object Interaction Diffusion via Relation Intervention

TL;DR

This paper proposes a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR), a cohesive diffusion model equipped with a relation intervention mechanism that enhances the spatial-temporal relations between humans and objects.

Abstract

This paper addresses new methodologies to deal with the challenging task of generating dynamic Human-Object Interactions from textual descriptions (Text2HOI). While most existing works assume interactions with limited body parts or static objects, our task involves addressing the variation in human motion, the diversity of object shapes, and the semantic vagueness of object motion simultaneously. To tackle this, we propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR). THOR is a cohesive diffusion model equipped with a relation intervention mechanism. In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion. This intervention enhances the spatial-temporal relations between humans and objects, with human-centric interaction representation providing additional guidance for synthesizing consistent motion from text. To achieve more reasonable and realistic results, interaction losses is introduced at different levels of motion granularity. Moreover, we construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset. Both quantitative and qualitative experiments demonstrate the effectiveness of our proposed model.

Figures (10)

• Figure 1: A novel task of generating 3D Human-Object Interaction based on a Text prompt (Text2HOI), reflecting When, Where, and How humans interact with the object.
• Figure 2: The overview of our model THOR, designed to address a novel task of generating human-object interactions from textual descriptions (Text2HOI). The key innovation lies in leveraging the human-object spatial relations to further intervene the object motion and then diffuse back to benefit the whole Text2HOI diffusion framework.
• Figure 3: Qualitative comparison for Text2HOI generation. Artifacts are highlighted in red circle. Our model, THOR, can generate realistic and plausible human-object interaction in response to the textual guidance. Through the intervention mechanism and two interaction losses, it corrects the drifting object motion and alleviate the implausible human-object spatial relations.
• Figure 4: Qualitative results of our model for Text2HOI generation. Our model can generate human-object interactions aligned with the text description involving static and dynamic objects with diverse categories and shapes.
• Figure 5: User study on Text-BEHAVE test set, where 'w/o' indicates 'without' and 'GT' indicates ground truth.