ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion

TL;DR

ManiDext presents a diffusion-based framework that generates dexterous, bimanual hand manipulations conditioned solely on 3D object trajectories. It introduces a continuous correspondence embedding to model dense hand-object contacts and a residual-guided diffusion process that merges generation and refinement within a single framework. The method operates in two stages—first predicting object-side contact maps and embeddings, then guiding hand pose synthesis with residuals in a canonical object-centric space—achieving state-of-the-art results on ARCTIC, GRAB, and HOI4D while maintaining efficiency. These advances improve physical plausibility, temporal coherence, and diversity of hand-object interactions, with broad implications for virtual humans and embodied AI.

Abstract

Dynamic and dexterous manipulation of objects presents a complex challenge, requiring the synchronization of hand motions with the trajectories of objects to achieve seamless and physically plausible interactions. In this work, we introduce ManiDext, a unified hierarchical diffusion-based framework for generating hand manipulation and grasp poses based on 3D object trajectories. Our key insight is that accurately modeling the contact correspondences between objects and hands during interactions is crucial. Therefore, we propose a continuous correspondence embedding representation that specifies detailed hand correspondences at the vertex level between the object and the hand. This embedding is optimized directly on the hand mesh in a self-supervised manner, with the distance between embeddings reflecting the geodesic distance. Our framework first generates contact maps and correspondence embeddings on the object's surface. Based on these fine-grained correspondences, we introduce a novel approach that integrates the iterative refinement process into the diffusion process during the second stage of hand pose generation. At each step of the denoising process, we incorporate the current hand pose residual as a refinement target into the network, guiding the network to correct inaccurate hand poses. Introducing residuals into each denoising step inherently aligns with traditional optimization process, effectively merging generation and refinement into a single unified framework. Extensive experiments demonstrate that our approach can generate physically plausible and highly realistic motions for various tasks, including single and bimanual hand grasping as well as manipulating both rigid and articulated objects. Code will be available for research purposes.

Figures (12)

• Figure 1: We present ManiDext, which takes a sequence of object motions as input and generates dexterous bimanual hand manipulations. The motions are depicted through snapshots at several key frames. Our hierarchical, diffusion-based pipeline first generates contact probability maps and continuous correspondence maps on the object's surface. These contact details then guide the subsequent stage of hand pose generation.
• Figure 2: Method Overview. Given a sequence of object motion trajectory, we adopt a hierarchical diffusion-based framework to gradually generate the hand poses that manipulate the object. First, we generate contact information on the object's surface, which includes a contact probability map and a continuous correspondence embedding map. These information provides dense correspondence, allow us to compute the geometric residual error at each diffusion timestep $T$. Subsequently, we use the generated contact information and the computed residual error as conditions to generate the manipulation hand poses.
• Figure 3: Illustration of differential object-centric motion modeling. (a) shows the motion modeling in the world coordinate system. The blue shows the object's position and orientation during data collection, while the green represents the same relative motion from a different orientation. (b) shows the hand-object modeling in object's canonical coordinate system, eliminating global information, shown in orange.
• Figure 4: (a) Visualization of the optimized continuous surface embedding of the hand in three dimensions, represented with colors. (b) Visualization of the embedding distances between a query vertex and all other vertices.
• Figure 5: Illustration of contact probability and correspondence embeddings defined on both the object and hand surfaces.