InterPose: Learning to Generate Human-Object Interactions from Large-Scale Web Videos

TL;DR

InterPose tackles realistic 3D human–object interaction generation by compiling a large-scale, automatically annotated dataset of 3D motions and captions from web videos. It combines a data-collection pipeline with 3D pose estimation and vision–language tools, enabling robust, zero-shot HOI generation. The work demonstrates substantial improvements to controllable motion models and introduces HOI-Agent, a zero-shot HOI system for complex scenes. Together, InterPose and HOI-Agent offer a scalable resource and a practical framework for HOI synthesis with broad implications for graphics, robotics, and embodied AI.

Abstract

Human motion generation has shown great advances thanks to the recent diffusion models trained on large-scale motion capture data. Most of existing works, however, currently target animation of isolated people in empty scenes. Meanwhile, synthesizing realistic human-object interactions in complex 3D scenes remains a critical challenge in computer graphics and robotics. One obstacle towards generating versatile high-fidelity human-object interactions is the lack of large-scale datasets with diverse object manipulations. Indeed, existing motion capture data is typically restricted to single people and manipulations of limited sets of objects. To address this issue, we propose an automatic motion extraction pipeline and use it to collect interaction-rich human motions. Our new dataset InterPose contains 73.8K sequences of 3D human motions and corresponding text captions automatically obtained from 45.8K videos with human-object interactions. We perform extensive experiments and demonstrate InterPose to bring significant improvements to state-of-the-art methods for human motion generation. Moreover, using InterPose we develop an LLM-based agent enabling zero-shot animation of people interacting with diverse objects and scenes.

Figures (8)

• Figure 1: Learning from online videos enables generation of complex human motions. (a) Our InterPose dataset is obtained from videos with varying scenes and activities as well as diverse human-object interactions. (b) Our HOI-Agent deploys InterPose for training and enables zero-shot generation of collision-free navigation, human-object interactions and multi-person collaboration in complex 3D scenes.
• Figure 2: Overview of data collection for the InterPose dataset. Our framework contains a module for collecting interaction-rich videos (left) and a module for automatic extraction of 3D human motions and corresponding text captions (right).
• Figure 3: Examples from InterPose. Our 3D human motion data originates from diverse interaction scenarios including working actions, sports activities, indoor and outdoor scenes. All motion sequences are annotated with action, object label and a detailed textual description.
• Figure 4: Impact of dataset size. We train OmniControl xie2023omnicontrol on HumanML3D guo2022generating dataset and subsets of InterPose of different sizes. The subsets are composed of 0%, 10%, 20%, 50%, and 100% of InterPose training set. All the models are evaluated on HumanML3D guo2022generating, OMOMO li2023object and BEHAVE bhatnagar22behave test sets. We report Traj 0.5 and MPJPE on pelvis control and hands control setting for all the models.
• Figure 5: Visual comparison of CHOIS li2023controllable, MaskedMimic tessler2024maskedmimic and Ours when evaluated on OMOMO li2023object and BEHAVE bhatnagar22behave datasets. Note that original MaskedMimic tessler2024maskedmimic is trained on AMASS mahmood2019amass dataset and Ours denotes MaskedMimic trained on the combination of InterPose and AMASS mahmood2019amass. Red boxes highlight examples of unrealistic human action, unrealistic object motion or interaction.