OccRobNet : Occlusion Robust Network for Accurate 3D Interacting Hand-Object Pose Estimation

TL;DR

Occlusion poses a major challenge for 3D hand pose estimation in interacting and hand-object scenarios. The authors introduce OccRobNet, a DETR-inspired, occlusion-robust architecture that combines a CNN backbone with a Contextual Information Enhancement Transformer (CIET) and a dual attention mechanism (sigmoid and softmax) to identify joints and hand identity even when occluded, followed by cross-attention-based pose estimation for hands and objects. The approach achieves state-of-the-art results on InterHand2.6M, HO-3D, and H2O3D datasets, with ablations showing the beneficial impact of CIET and the sigmoid attention module. This work advances reliable 3D hand–object interaction understanding under occlusion, with potential for real-time, RGB-based deployment.

Abstract

Occlusion is one of the challenging issues when estimating 3D hand pose. This problem becomes more prominent when hand interacts with an object or two hands are involved. In the past works, much attention has not been given to these occluded regions. But these regions contain important and beneficial information that is vital for 3D hand pose estimation. Thus, in this paper, we propose an occlusion robust and accurate method for the estimation of 3D hand-object pose from the input RGB image. Our method includes first localising the hand joints using a CNN based model and then refining them by extracting contextual information. The self attention transformer then identifies the specific joints along with the hand identity. This helps the model to identify the hand belongingness of a particular joint which helps to detect the joint even in the occluded region. Further, these joints with hand identity are then used to estimate the pose using cross attention mechanism. Thus, by identifying the joints in the occluded region, the obtained network becomes robust to occlusion. Hence, this network achieves state-of-the-art results when evaluated on the InterHand2.6M, HO3D and H$_2$O3D datasets.

Figures (4)

• Figure 1: Overview of our method. We first estimates 1D heatmaps from RGB hand image and hand-object segmentation mask \ref{['sec:1']}, using ResNet50 network. From the feature maps $F$, the CIET module, which is a transformer module, creates context-aware features which are the refined object and hand features \ref{['sec:2']}. Then, transformer encoder network is utilized to give attention to hand and object features using sigmoid and softmax attentions. The decoder cross-attention module finally predicts the hand pose \ref{['sec:3']}.
• Figure 2: The detailed structure of our Contextual Information Enhancement Transformer (CIET) block. $\star$ denotes the local matrix multiplication and $\theta$ and $\delta$, are used to denote the two 1 $\times$ 1 convolution.
• Figure 3: The overall structure that combines the softmax attention with the sigmoid attention for the same input query- key pairs.
• Figure 4: Qualitative results for our method on the HO-3D. Here, (a), (b) shows Joint projections, and (c), (d) represents Hand Segmentation.