PartHOI: Part-based Hand-Object Interaction Transfer via Generalized Cylinders

TL;DR

This work addresses the scarcity of high-quality HOI data and the challenge of transferring hand grasps across object categories. It introduces PartHOI, a part-based HOI transfer pipeline that uses generalized cylinders to represent object parts, enabling size-invariant contact-map transfer and subsequent hand pose optimization. The approach yields superior cross-category transfer fidelity and stability compared to baselines, demonstrated on a new dataset and supported by perceptual studies. By enabling robust, cross-category hand-object transfers without extensive training data, PartHOI facilitates data-efficient HOI propagation and broader applicability in realistic 3D scene understanding.

Abstract

Learning-based methods to understand and model hand-object interactions (HOI) require a large amount of high-quality HOI data. One way to create HOI data is to transfer hand poses from a source object to another based on the objects' geometry. However, current methods for transferring hand poses between objects rely on shape matching, limiting the ability to transfer poses across different categories due to differences in their shapes and sizes. We observe that HOI often involves specific semantic parts of objects, which often have more consistent shapes across categories. In addition, constructing size-invariant correspondences between these parts is important for cross-category transfer. Based on these insights, we introduce a novel method PartHOI for part-based HOI transfer. Using a generalized cylinder representation to parameterize an object parts' geometry, PartHOI establishes a robust geometric correspondence between object parts, and enables the transfer of contact points. Given the transferred points, we optimize a hand pose to fit the target object well. Qualitative and quantitative results demonstrate that our method can generalize HOI transfers well even for cross-category objects, and produce high-fidelity results that are superior to the existing methods.

Figures (12)

• Figure 1: We introduce PartHOI, a pipeline for hand pose transfer, based on object parts represented by generalized cylinders. We transfer a hand pose from an existing hand-object interaction to a new object. Our work can transfer plausible hand grasps between cross-category objects, even with a large geometry difference (a, b) or a large size difference (c). In contrast, the existing work OakInk, its part-based version and, part-based direct transfer via correspondence would easily lose hand contact and produce unnatural hand poses if there are significant differences between the two objects.
• Figure 2: Examples of contact map transfers based on (a) global-based shape interpolation and (b) part-based shape interpolation.
• Figure 3: The pipeline of our PartHOI method. (a) We first model a given source and target object part using a generalized cylinder representation. (b) Next, we transfer the contact map from the source part to the target one via GC-based correspondence. (c) Following the target contact map, we adopt a hand optimizer similar to OakInk to optimize a hand pose initialized from the source hand pose.
• Figure 4: Abstract and parameterize object parts represented as GCs.
• Figure 5: Given a segmented object part (highlighted in cyan in (a)), our method first abstracts and parameterizes the shape into a GC (b), which represents each surface point $p$ via a polar coordinate system (c).