SPLite Hand: Sparsity-Aware Lightweight 3D Hand Pose Estimation

TL;DR

This work tackles real-time 3D hand pose estimation on edge devices by integrating sparse convolution on a lightweight ResNet-18 with a novel SPLite decoder in a 3-stage pipeline (early fusion, 2D encoding, and 3D decoding). It introduces a trainable 2D-to-3D lifting mechanism and a sparsity-aware decoding scheme, delivering up to a 3× speed-up on Raspberry Pi 5 while preserving accuracy (PA-MPJPE around $9.1$ mm on FreiHAND after quantization-aware training and memory usage reduced by $75\%$ to $18$ MB). Key innovations include an edge-based fusion strategy that induces high sparsity ($\sim$89.2\%) and a SPLite decoder that conpares favorably to MobRecon in both speed and resource use (with a $28\%$ improvement over baseline and $65\%$ over a ResNet-18 variant). The approach is validated on multimodal hand-object datasets, demonstrating competitive accuracy with significantly improved computational efficiency, enabling practical edge deployment. The work also contributes a multimodal HOI dataset with diverse object categories and interactions to support robust evaluation. Overall, the framework enables real-time, energy-efficient hand pose estimation suitable for AR/VR and robotics on resource-limited devices.

Abstract

With the increasing ubiquity of AR/VR devices, the deployment of deep learning models on edge devices has become a critical challenge. These devices require real-time inference, low power consumption, and minimal latency. Many framework designers face the conundrum of balancing efficiency and performance. We design a light framework that adopts an encoder-decoder architecture and introduces several key contributions aimed at improving both efficiency and accuracy. We apply sparse convolution on a ResNet-18 backbone to exploit the inherent sparsity in hand pose images, achieving a 42% end-to-end efficiency improvement. Moreover, we propose our SPLite decoder. This new architecture significantly boosts the decoding process's frame rate by 3.1x on the Raspberry Pi 5, while maintaining accuracy on par. To further optimize performance, we apply quantization-aware training, reducing memory usage while preserving accuracy (PA-MPJPE increases only marginally from 9.0 mm to 9.1 mm on FreiHAND). Overall, our system achieves a 2.98x speed-up on a Raspberry Pi 5 CPU (BCM2712 quad-core Arm A76 processor). Our method is also evaluated on compound benchmark datasets, demonstrating comparable accuracy to state-of-the-art approaches while significantly enhancing computational efficiency.

Figures (5)

• Figure 1: our efficient pipeline begins with an RGB image being converted to grayscale, from which a proprietary algorithm extracts an edge image. A customized backbone called Sparse ResNet-18 then uses sparse convolutional blocks to generate 2D feature maps and predicted depth values. To improve the robustness of the depth prediction, a second data stream is introduced during training to account for perspective ambiguity. Finally, a calibrated camera intrinsic transformation module converts the 2D feature maps from camera coordinates to real-world coordinates for the lightweight SPLite decoder, which ultimately predicts a 3D hand mesh.
• Figure 2: Qualitative comparison on in-the-wild samples. Each row shows an input RGB image (left), followed by our predicted 3D joints, our reconstructed 3D mesh, and results from MobileHandlim2020mobilehand and Mobreconchen2022mobreconmobilefriendlyhandmesh. Our method produces more accurate and natural hand poses, preserving fine articulation and finger alignment, while competing methods exhibit noticeable distortions or misalignments, particularly in challenging poses and occlusions.
• Figure 3: Qualitative comparison on in-the-wild samples. Each row shows an input edge image (left). Our method produces accurate and natural hand poses, while competing methods lim2020mobilehandchen2022mobreconmobilefriendlyhandmesh demonstrate unreasonable distortions and fail to produce a coherent hand structure on this modality.
• Figure 4: The dataset is captured in real-world scenarios using a proprietary sensor, yielding 128x128 resolution images synchronized with two Intel RealSense cameras from different perspectives. The dataset includes four distinct scenarios, as shown in Figure \ref{['fig:dataset']}.100 genres of object categories and corresponding action manipulation respectively. The ground-truth keypoints and meshes were labeled semi-manually.
• Figure 5: Illustration of our dataset, which includes object and motion categories—each with 25 unique objects and corresponding manipulation actions. We designed four common daily scenarios in the dataset to aid in robot model training. Images were captured using two Intel RealSense cameras and one proprietary camera. Ground-truth keypoints and meshes were semi-manually labeled, as detailed in the Appendix.