F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions

TL;DR

This work identifies a gap in fine-grained semantic alignment for 3D human-object interactions (HOI) and introduces Semantic-HOI, a dataset with over $20{,}441$ state-level HOI pairs and inter-state movements. It proposes F-HOI, a unified multimodal framework that uses multi-modal encoders and a large language model to jointly tackle four HOI tasks—Understanding, Reasoning, Generation, and Reconstruction—by aligning HOI states with detailed textual descriptions. Through pretraining and multi-task instruction tuning, F-HOI demonstrates improved performance over a strong baseline across all tasks, supported by ablations that validate design choices such as offset regression and cross-modal alignments. The work lays a foundation for fine-grained, state-level HOI modeling and points to future directions in flexible inputs, larger-scale data, and alternative architectures to further enhance generation and reasoning in HOI sequences.

Abstract

Existing 3D human object interaction (HOI) datasets and models simply align global descriptions with the long HOI sequence, while lacking a detailed understanding of intermediate states and the transitions between states. In this paper, we argue that fine-grained semantic alignment, which utilizes state-level descriptions, offers a promising paradigm for learning semantically rich HOI representations. To achieve this, we introduce Semantic-HOI, a new dataset comprising over 20K paired HOI states with fine-grained descriptions for each HOI state and the body movements that happen between two consecutive states. Leveraging the proposed dataset, we design three state-level HOI tasks to accomplish fine-grained semantic alignment within the HOI sequence. Additionally, we propose a unified model called F-HOI, designed to leverage multimodal instructions and empower the Multi-modal Large Language Model to efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It employs a unified task formulation that supports the use of versatile multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and linguistic spaces. (3) It utilizes fine-grained textual supervision for direct optimization, avoiding intricate modeling of HOI states. Extensive experiments reveal that F-HOI effectively aligns HOI states with fine-grained semantic descriptions, adeptly tackling understanding, reasoning, generation, and reconstruction tasks.

Figures (7)

• Figure 1: Illustration of three state-level tasks to achieve fine-grained semantic alignment.
• Figure 2: Illustration of the annotations in Semantic-HOI for a paired HOI sample.
• Figure 3: Input and output definitions for each task.
• Figure 4: Overview of our F-HOI framework, which contains the three components: multi-modal encoders, a large language model, and task-specific projectors. Based on different task instructions, F-HOI could support multi-modal inputs and complete diverse HOI tasks, covering understanding, reasoning, generation, and reconstruction tasks.
• Figure 5: Qualitative results of F-HOI on generation task.