Hoi! -- A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation

TL;DR

Hoi! introduces a multimodal, force-grounded dataset that pairs human and robot demonstrations of everyday articulated objects across multiple viewpoints. It combines RGB-D, force-torque, and GelSight tactile sensing with four embodiments and precise scene geometry, enabling grounded cross-embodiment transfer and force-aware perception. The paper benchmarks articulation estimation, tactile force estimation, and visual force estimation, revealing significant domain gaps and challenges in real-world, cluttered settings. Overall, Hoi! provides a rich resource to study how vision, touch, and physics interact in real-world manipulation and to train models that generalize across embodiments.

Abstract

We present a dataset for force-grounded, cross-view articulated manipulation that couples what is seen with what is done and what is felt during real human interaction. The dataset contains 3048 sequences across 381 articulated objects in 38 environments. Each object is operated under four embodiments - (i) human hand, (ii) human hand with a wrist-mounted camera, (iii) handheld UMI gripper, and (iv) a custom Hoi! gripper - where the tool embodiment provides synchronized end-effector forces and tactile sensing. Our dataset offers a holistic view of interaction understanding from video, enabling researchers to evaluate how well methods transfer between human and robotic viewpoints, but also investigate underexplored modalities such as force sensing and prediction.

Figures (13)

• Figure 1: Overview of the Hoi! Dataset: A multimodal dataset for force-grounded, cross-view articulated manipulation in wild indoor environments. The dataset captures human interactions with common articulated objects (drawers, doors, fridges, dishwashers) with synchronized RGB, depth, force, tactile sensing, and multi-view videos from egocentric and exocentric perspectives. Each interaction is annotated with articulation parameters (e.g., opening angles, displacements, peak forces), supporting research on multimodal perception, manipulation learning, and embodied reasoning.
• Figure 2: Locations of the Hoi! dataset. A diverse collection of real-world indoor environments featuring kitchens, bathrooms, offices, and living spaces, were each has RGB-D sequences, GT, panoramic images, and various articulated objects that have interactions with multiple grippers and users.
• Figure 3: Hoi! Gripper. The 2-finger parallel gripper is operated through the load cell, where the measured load is translated into gripping force. Interaction force and tactile contact pressure are measured through the Digit and Force-Torque sensors respectively. Aria Glasses and a stereo camera provide pose estimation and wrist-view observations. We will release the design as open source.
• Figure 4: Example of the measured interaction forces for several articulated elements. Each curve corresponds to a different component (highlighted in matching colors below), illustrating how force magnitudes vary across types of articulated parts.
• Figure 5: Distribution of environments and articulated interaction categories in the Hoi! dataset. The bar chart depicts the relative frequency of human interactions across articulated categories, while the inset pie chart summarizes the proportion of environments involved in the interactions.