DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors

TL;DR

This work introduces a dual implicit-explicit representation of a skinned mesh, combining (implicit) neural radiance fields (NeRFs) with (explicit) skeleton-driven mesh articulation, enabling a 3D human model to realistically interact with any given object based on a textual description.

Abstract

We present DreamHOI, a novel method for zero-shot synthesis of human-object interactions (HOIs), enabling a 3D human model to realistically interact with any given object based on a textual description. This task is complicated by the varying categories and geometries of real-world objects and the scarcity of datasets encompassing diverse HOIs. To circumvent the need for extensive data, we leverage text-to-image diffusion models trained on billions of image-caption pairs. We optimize the articulation of a skinned human mesh using Score Distillation Sampling (SDS) gradients obtained from these models, which predict image-space edits. However, directly backpropagating image-space gradients into complex articulation parameters is ineffective due to the local nature of such gradients. To overcome this, we introduce a dual implicit-explicit representation of a skinned mesh, combining (implicit) neural radiance fields (NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization, we transition between implicit and explicit forms, grounding the NeRF generation while refining the mesh articulation. We validate our approach through extensive experiments, demonstrating its effectiveness in generating realistic HOIs.

Figures (11)

• Figure 1: Generated Human-Object Interactions by DreamHOI. (a-c) DreamHOI takes as inputs a skinned human model, an object mesh, and a textual description of the intended interaction between them. It then poses the human model to create realistic interactions. (b) Given the same interaction description, the generated pose naturally conforms to the intricacies of the input object to be interacted with. (c) Given a fixed object, the generated poses vary faithfully to different intended interactions.
• Figure 2: Overview of DreamHOI. Our method takes a human identity (in the form of a skinned body mesh) and an object mesh $M_{\text{Obj}}$ (e.g., a 3D chair), together with their intended interaction (as a textual prompt, e.g., "sit"), as input. It first fits a NeRF $f_{\theta_0}$ for the human using a mixture of diffusion guidance and regularizers, and then estimates its pose $\xi_0$. The posed human mesh $M_{\xi_t}$ is used to re-initialize and further optimize the NeRF $f_{\theta_{t}}$, for iterations $t\leq T$. The final output is the posed human $M_{\xi_T}$ at the last iteration. See \ref{['sec:methods']}.
• Figure 3: Additional Results. We demonstrate DreamHOI's ability to control the pose based on different textual conditions.
• Figure 4: Comparison with baselines: (third row) using DreamFusion to optimize a NeRF with the given object mesh inserted; (last row) using DreamFusion to optimize the pose parameters of the skinned human mesh directly. See \ref{['sec:comparisons']} for discussions.
• Figure 5: Ablations. We visualize output of first-round NeRF optimization (i.e., $f_{\theta_0}$). (Row 2 vs. row 3) Our regularizers (\ref{['sec:regularizers']}) effectively prevent the NeRF from encroaching upon and overriding the mesh. (Row 2 vs. row 4) The human-only SDS motivates a complete human while MVDream enhances view consistency (note the TV and wardrobe in row 4 have multiple faces).