AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation

TL;DR

Av AvatarGO is introduced, a novel framework designed to generate animatable 4D HOI scenes directly from textual inputs that not only generates coherent compositional motions, but also exhibits greater robustness in handling penetration issues.

Abstract

Recent advancements in diffusion models have led to significant improvements in the generation and animation of 4D full-body human-object interactions (HOI). Nevertheless, existing methods primarily focus on SMPL-based motion generation, which is limited by the scarcity of realistic large-scale interaction data. This constraint affects their ability to create everyday HOI scenes. This paper addresses this challenge using a zero-shot approach with a pre-trained diffusion model. Despite this potential, achieving our goals is difficult due to the diffusion model's lack of understanding of ''where'' and ''how'' objects interact with the human body. To tackle these issues, we introduce AvatarGO, a novel framework designed to generate animatable 4D HOI scenes directly from textual inputs. Specifically, 1) for the ''where'' challenge, we propose LLM-guided contact retargeting, which employs Lang-SAM to identify the contact body part from text prompts, ensuring precise representation of human-object spatial relations. 2) For the ''how'' challenge, we introduce correspondence-aware motion optimization that constructs motion fields for both human and object models using the linear blend skinning function from SMPL-X. Our framework not only generates coherent compositional motions, but also exhibits greater robustness in handling penetration issues. Extensive experiments with existing methods validate AvatarGO's superior generation and animation capabilities on a variety of human-object pairs and diverse poses. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.

Figures (15)

• Figure 1: Examples of 4D animation results obtained via AvatarGO. AvatarGO effectively produces diverse human-object compositions with correct spatial correlations and contact areas. It achieves joint animation of humans and objects while avoiding penetration issues.
• Figure 1: Quantitative Evaluation.
• Figure 2: Overview of AvatarGO. AvatarGO takes the text prompts as input to generate 4D avatars with object interactions. At the core of our network are: 1) Text-driven 3D human and object composition that employs large language models to retarget the contact areas from texts and spatial-aware SDS to composite the 3D models. 2) Correspondence-aware motion optimization which jointly optimizes the animation for humans and objects. It effectively maintains the spatial correspondence during animation, addressing the penetration issues.
• Figure 3: Comparisons on 3D compositional generations.
• Figure 4: Comparisons on 4D avatar animation with object interactions. '$*$' indicates that HumanGaussian directly employs the SMPL LBS function for animation.