TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures

TL;DR

This work introduces TeHOR, a framework that leverages text descriptions of human-object interactions to enforce semantic alignment between the 3D reconstruction and its textual cues, enabling reasoning over a wider spectrum of interactions, including non-contact cases.

Abstract

Joint reconstruction of 3D human and object from a single image is an active research area, with pivotal applications in robotics and digital content creation. Despite recent advances, existing approaches suffer from two fundamental limitations. First, their reconstructions rely heavily on physical contact information, which inherently cannot capture non-contact human-object interactions, such as gazing at or pointing toward an object. Second, the reconstruction process is primarily driven by local geometric proximity, neglecting the human and object appearances that provide global context crucial for understanding holistic interactions. To address these issues, we introduce TeHOR, a framework built upon two core designs. First, beyond contact information, our framework leverages text descriptions of human-object interactions to enforce semantic alignment between the 3D reconstruction and its textual cues, enabling reasoning over a wider spectrum of interactions, including non-contact cases. Second, we incorporate appearance cues of the 3D human and object into the alignment process to capture holistic contextual information, thereby ensuring visually plausible reconstructions. As a result, our framework produces accurate and semantically coherent reconstructions, achieving state-of-the-art performance.

Figures (18)

• Figure 1: TeHOR. Given a single image, our framework jointly reconstructs textured 3D human and object by capturing their holistic and semantic interactions using text descriptions.
• Figure 2: Limitations of existing reconstruction methods. Previous methods suffer from over-reliance on contact information and neglect the global interaction context, leading to implausible reconstructions. In contrast, TeHOR produces accurate and plausible 3D reconstructions by leveraging holistic and semantic guidance from text descriptions.
• Figure 3: Overall pipeline of TeHOR. Given an input image, our framework initially reconstructs a 3D human, a 3D object, and a 2D background. Then, the initially reconstructed 3D human and object are jointly optimized using three core loss functions: reconstruction loss, appearance loss, and contact loss, to ensure accurate and semantically plausible human-object interaction.
• Figure 4: Gaussians-to-mesh conversion process.
• Figure 5: Optimization results of TeHOR. Our text-guided optimization accurately refines the 3D human and object by utilizing their corresponding text descriptions.