Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimation

TL;DR

This work tackles the challenge of hand pose estimation by rethinking representation learning through transformation isomorphism: aligning transformations across image, latent, and pose spaces. TI-Net learns a transformation-consistent latent space using lightweight latent transformers and rotation-embedding to ensure the latent features reflect pose-related, low-level information. Through pretraining that combines reconstruction with ordinary and secondary transformation constraints, TI-Net achieves state-of-the-art PA-MPJPE on DexYCB and strong results on InterHand2.6M, while also offering faster convergence and easy integration with existing pose estimation frameworks. The approach demonstrates that transformation-consistent, compact latent representations can significantly improve regression performance in vision tasks and invites extension to other regression problems beyond hand pose estimation.

Abstract

Vision-based regression tasks, such as hand pose estimation, have achieved higher accuracy and faster convergence through representation learning. However, existing representation learning methods often encounter the following issues: the high semantic level of features extracted from images is inadequate for regressing low-level information, and the extracted features include task-irrelevant information, reducing their compactness and interfering with regression tasks. To address these challenges, we propose TI-Net, a highly versatile visual Network backbone designed to construct a Transformation Isomorphic latent space. Specifically, we employ linear transformations to model geometric transformations in the latent space and ensure that {\rm TI-Net} aligns them with those in the image space. This ensures that the latent features capture compact, low-level information beneficial for pose estimation tasks. We evaluated TI-Net on the hand pose estimation task to demonstrate the network's superiority. On the DexYCB dataset, TI-Net achieved a 10% improvement in the PA-MPJPE metric compared to specialized state-of-the-art (SOTA) hand pose estimation methods. Our code will be released in the future.

Figures (5)

• Figure 1: In the pretraining phase, (I) Contrastive learning approaches attract positive pairs and repel negative pairs.chen_simclr_2020le-khac_contrastive_2020 (II) Masked image modeling approach reconstructs the image from the embedding of the original one.he_masked_2021li_masked_2024 (III) TI-Net ensures that the transformation relationships in the image space also hold in the latent space, as does the combined result of transformations. We refer to this property as "transformation isomorphism."
• Figure 2: Overview of transformation isomorphism. Left: The relationships among three transformations in the image space: horizontal flip, rotation, and the horizontal flip + rotation. Any two of these transformations can be composed to form another transformation, and the rotation inherently includes the identity transformation. Right: In the pose space, there are three transformations that correspond exactly to the three transformations in the image space, and they satisfy the same combination rules. We refer to this perfect correspondence as transformation isomorphism. Center: TI-Net ensures that there exists transformations in latent space that correspond to the ones in image space. Due to the equivalence property of the isomorphism, the transformations in the latent space also correspond to those in the pose space, and satisfying the same combination rules.
• Figure 3: Simplified overview of pretraining phase. Weights of latent transformation are updated jointly with TI-Net. We depict only one ordinary and one secondary constraint here for simplicity.
• Figure 4: Visualization comparison between our approach and SimCLRchen_simclr_2020 on DexYCBchao:cvpr2021_dexycb Our method exhibits better accuracy under occlusion scenes. TI-Net and SimCLRchen_simclr_2020 are finetuned on DexYCBchao:cvpr2021_dexycb under the same procedure and meta-parameters. GT standards for ground truth annotations. We adjusted the viewing direction for the best comparison.
• Figure 5: Comparison of MPJPE between our approach and SimCLRchen_simclr_2020 approach on every epoch on DexYCBchao:cvpr2021_dexycb, aligning all training setup the same. Our approach shows faster convergence and more stable training.