SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representation

TL;DR

SynHLMA addresses the challenge of generating realistic hand articulation during articulated object manipulation by grounding discrete HAOI representations in language. It introduces a multi-stage VQ-VAE to discretize per-frame hand-object interactions and an HAOI manipulation language model to produce complete manipulation sequences conditioned on natural language, trained with an articulation-aware loss. The paper contributes the HAOI-Lang dataset, a hierarchical tokenization scheme, and a two-stage language-model training paradigm, achieving state-of-the-art results on HAOI generation, prediction, and interpolation and enabling dexterous grasp transfer to robotics. This approach advances embodied AI by tightly coupling language-described manipulation with physically plausible articulated-object dynamics, with potential impact on robotic dexterity and language-grounded control.

Abstract

Generating hand grasps with language instructions is a widely studied topic that benefits from embodied AI and VR/AR applications. While transferring into hand articulatied object interaction (HAOI), the hand grasps synthesis requires not only object functionality but also long-term manipulation sequence along the object deformation. This paper proposes a novel HAOI sequence generation framework SynHLMA, to synthesize hand language manipulation for articulated objects. Given a complete point cloud of an articulated object, we utilize a discrete HAOI representation to model each hand object interaction frame. Along with the natural language embeddings, the representations are trained by an HAOI manipulation language model to align the grasping process with its language description in a shared representation space. A joint-aware loss is employed to ensure hand grasps follow the dynamic variations of articulated object joints. In this way, our SynHLMA achieves three typical hand manipulation tasks for articulated objects of HAOI generation, HAOI prediction and HAOI interpolation. We evaluate SynHLMA on our built HAOI-lang dataset and experimental results demonstrate the superior hand grasp sequence generation performance comparing with state-of-the-art. We also show a robotics grasp application that enables dexterous grasps execution from imitation learning using the manipulation sequence provided by our SynHLMA. Our codes and datasets will be made publicly available.

Figures (10)

• Figure 1: Given an articulated object shape and a natural language as instruction, our SynHLMA achieves three typical hand manipulation synthesis tasks: HAOI generation (up), HAOI prediction (middle) and HAOI interpolation (down) with their corresponding manipulation descriptions.
• Figure 2: Our SynHLMA pipeline, the upper blue dashed region illustrates the training process of our proposed Discrete Articulated Manipulation Representation model. The lower red region depicts the HAOI Manipulation Language Model, where the parameters of the Discrete Articulated Manipulation Representation are kept frozen during training.
• Figure 3: Synthesis HAOI data generation process. (a) grasp region perception. (b) manipulation sequence generation with RL. (c) language instruction generation with GPT-4. (d) large number of samples demonstration.
• Figure 4: HAOI Manipulation Language Model
• Figure 5: Qualitative results comparison between our SynHLMA and HOIGPT on three HAOI tasks of HAOI generation, HAOI prediction, and HAOI interpolation.