Learning Dense Hand Contact Estimation from Imbalanced Data

TL;DR

This work tackles dense hand contact estimation under severe data imbalance by introducing HACO, a transformer-based framework that leverages Balanced Contact Sampling (BCS) and Vertex-Level Class-Balanced (VCB) loss to learn from a large, diverse set of 14 datasets with $V=778$ MANO vertices. The model employs a ViT backbone with a dedicated contact token and multi-level supervision to predict dense contact maps while mitigating both class and spatial biases. Empirical results show HACO achieves state-of-the-art performance on dense hand contact estimation and yields tangible gains in downstream 3D grasp optimization and hand-object reconstruction. The approach offers practical impact for AR/VR, robotics, and behavior analysis by enabling more accurate, generalizable contact understanding from visual input.

Abstract

Hands are essential to human interaction, and exploring contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of regions are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact vertices. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes are available at https://github.com/dqj5182/HACO_RELEASE.

Figures (10)

• Figure 1: Two challenges for dense hand contact estimation in the wild. First, hand contact datasets suffer from class imbalance, as the majority of vertices contain no contact. Second, spatial imbalance arises because contact points are predominantly concentrated at the fingertips. Due to these issues, models trained on such data struggle to generalize to diverse contact patterns across the hand.
• Figure 2: Overall pipeline of HACO. Our method encodes input image as image tokens with a ViT backbone after patch embedding layers. Given the image tokens along with positional embeddings and a contact token, multiple layers of self-attention Transformer and cross-attention Transformer produce an output token. Lastly, the output token is passed through a linear layer and combined with the contact initialization, followed by a sigmoid activation to output the final hand contact prediction.
• Figure 3: Qualitative comparison of dense hand contact estimation with POSA hassan2021populating, BSTRO huang2022capturing, DECO tripathi2023deco on MOW cao2021reconstructing dataset. We highlight exemplar regions where HACO outperforms previous methods. Note that we only predict right hand contact.
• Figure 4: Qualitative comparison of 3D hand-object reconstruction on MOW cao2021reconstructing dataset.
• Figure A1: More examples of class imbalance between contact and non-contact on H2O kwon2021h2o, H2O3D hampali2022keypoint, HIC tzionas2016capturing, MOW cao2021reconstructing, ObMan hasson2019learning, PROX hassan2019resolving, RICH huang2022capturing, Hi4D yin2023hi4d.