GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency

TL;DR

GRIP tackles the challenge of generating realistic hand-object interaction poses given body and object motion by introducing two hand sensors (Ambient and Proximity) and a two-stage, feed-forward inference pipeline. An Arm Denoising Network (ANet) precedes a Consistency Network (CNet) that employs Latent Temporal Consistency (LTC) to produce temporally coherent hand poses, followed by a Refinement Network (RNet) that reduces penetrations via recalculated proximity cues. The approach is trained on GRAB and shown to generalize to unseen objects and other MoCap datasets, with strong quantitative and perceptual performance improvements across metrics such as $MPJPE$, $MPVPE$, IV, and CC. GRIP enables rapid hand-object synthesis for avatars and data augmentation, delivering realistic, temporally consistent dexterous interactions suitable for AR/VR, animation, and data generation tasks. Its combination of distance-based hand sensors, LTC-based latent consistency, and end-to-end learnable refinement offers a scalable alternative to optimization-based hand pose refinement.

Abstract

Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. Consequently, modeling realistic hand-object interactions, including the subtle motion of individual fingers, is critical for applications in computer graphics, computer vision, and mixed reality. Prior work on capturing and modeling humans interacting with objects in 3D focuses on the body and object motion, often ignoring hand pose. In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction. As a preliminary step before synthesizing the hand motion, we first use a network, ANet, to denoise the arm motion. Then, we leverage the spatio-temporal relationship between the body and the object to extract two types of novel temporal interaction cues, and use them in a two-stage inference pipeline to generate the hand motion. In the first stage, we introduce a new approach to enforce motion temporal consistency in the latent space (LTC), and generate consistent interaction motions. In the second stage, GRIP generates refined hand poses to avoid hand-object penetrations. Given sequences of noisy body and object motion, GRIP upgrades them to include hand-object interaction. Quantitative experiments and perceptual studies demonstrate that GRIP outperforms baseline methods and generalizes to unseen objects and motions from different motion-capture datasets.

Figures (15)

• Figure 1: GRIP generates realistic hand-object interaction poses (pink), given the easy-to-acquire body and object motion without fingers (blue) -- notice that the input hand pose is constant. GRIP animates the hands to be consistent with the body and object, producing realistic poses in various scenarios like pre-/post-grasp hand opening, and single or bi-manual grasps. It also works with various object shapes and sizes, and on different datasets like GRAB GRAB:2020 (left) and InterCap huang2022intercap (right).
• Figure 2: Overview of GRIP. We first denoise the arm motion using the ANet network. We then predict hand interaction motion in two stages: (CNet) Given the hand-object spatial features, extracted using our Hand Sensors, body pose and trajectories in two consecutive frames, CNet predicts both left- and right-hand poses. (RNet) Based on the predicted hand poses, we recompute the Proximity Sensor feature and refine the hand poses with RNet to enhance interaction accuracy and reduce possible penetrations.
• Figure 3: Visualization of our Hand Sensors (only right-hand for simplicity). (A)Ambient Sensor points (blue) and their computed distances to the closest object points (red). This captures the object geometry and distance to the hands. (B)Proximity Sensor feature computation for CNet's inputs with mean-hand pose initialization. (C) Recomputing the Proximity Sensor values for RNet, using the hand poses generated by CNet. Note that the corresponding points on the object change for each finger compared to (B).
• Figure 4: CNet Architecture. We propose the LTC algorithm that enforces consistency between two successive frames in the latent space (see \ref{['sec:Cnet']} for more details).
• Figure 5: Comparing CNet and RNet generated grasps. Results show that RNet effectively refines the penetration and "non-contact" artifacts (red circles) of the CNet results.