HandMCM: Multi-modal Point Cloud-based Correspondence State Space Model for 3D Hand Pose Estimation

TL;DR

HandMCM tackles occlusion-robust 3D hand pose estimation by introducing a novel state-space based correspondence mechanism, the Mamba, within a multi-modal RGB-D pipeline. It combines a Multi-Modal Super Point Encoder, a Keypoint Token Extraction stage, and a Bi-directional Correspondence SSM (BiGS) with local token injection and filtering to model dynamic kinematic relationships among hand joints. The approach achieves state-of-the-art mean keypoint errors on NYU ($7.06$ mm) and the hand-object benchmarks DexYCB ($1.71$ cm) and HO3D ($1.71$ cm), while maintaining real-timeish performance (~$15.4$ ms/frame) on an RTX $4090$. These results demonstrate the practical potential of integrating SSM-based correspondence with rich multi-modal representations for reliable 3D hand pose estimation in challenging, occlusion-heavy settings, with code publicly available.

Abstract

3D hand pose estimation that involves accurate estimation of 3D human hand keypoint locations is crucial for many human-computer interaction applications such as augmented reality. However, this task poses significant challenges due to self-occlusion of the hands and occlusions caused by interactions with objects. In this paper, we propose HandMCM to address these challenges. Our HandMCM is a novel method based on the powerful state space model (Mamba). By incorporating modules for local information injection/filtering and correspondence modeling, the proposed correspondence Mamba effectively learns the highly dynamic kinematic topology of keypoints across various occlusion scenarios. Moreover, by integrating multi-modal image features, we enhance the robustness and representational capacity of the input, leading to more accurate hand pose estimation. Empirical evaluations on three benchmark datasets demonstrate that our model significantly outperforms current state-of-the-art methods, particularly in challenging scenarios involving severe occlusions. These results highlight the potential of our approach to advance the accuracy and reliability of 3D hand pose estimation in practical applications.

Figures (4)

• Figure 1: Illustration of our HandMCM concept. The model extracts multi-modal 3D point features from input depth images, RGB images, and corresponding point clouds. The Mamba architecture directly explores geometric and kinematic correspondences in 3D point space to accurately recover hand poses.
• Figure 2: The HandMCM architecture. HandMCM takes as input an RGB image, a 2D depth image. The multi-modal super point encoder utilizes a PointNet-based local 3D encoder and two 2D autoencoders to extract local 3D point features and local 2D features from the RGB and depth inputs, respectively. The encoder then aggregates these projected 2D features with the 3D features, forming multi-modal super point features. Keypoint tokens are derived from the super points and fed into the novel correspondence Mamba to accurately estimate 3D keypoint coordinates.
• Figure 3: Qualitative results of the HandMCM model on the NYU dataset. 3D points are colored in the figure to distinctly illustrate occlusions. The ground truth joint coordinates are displayed in black, the results of the previous state-of-the-art model, HandDAGT cheng2024handdagt, are shown in orange (top), the results of Hamba dong2024hamba are shown in blue (bottom), and the estimated joint coordinates produced by our model are depicted in red. Best view in color.
• Figure 4: Qualitative results of HandMCM on the DexYCB datasets illustrating different object occlusions (left), view-angle-dependent occlusions (middle) and self-occlusions (right). Input 3D points and output keypoint coordinates are depicted within the same frame (third rows) and rotated at top-down view angles in order to clearly view occlusions from above as shown in the figure. Ground truth is shown in black and the estimated joint coordinates of our model are shown in colors. Best view in color.