Optimizing Hand Region Detection in MediaPipe Holistic Full-Body Pose Estimation to Improve Accuracy and Avoid Downstream Errors
Amit Moryossef
TL;DR
This work targets inaccuracies in MediaPipe Holistic's hand ROI predictions that arise with non-ideal hand orientations, addressing potential downstream errors in sign-language recognition. It proposes a data-driven enhancement that expands the input feature set to include additional hand keypoints and depth information, exploring a Kolmogorov-Arnold Network and a lightweight MLP-based solution to predict ROI center, size, and rotation. Experimental results on the Panoptic Hand DB show IoU improvement from 57% to 63% and a higher minimum IoU, with center and size predictions improved at the cost of rotation accuracy, highlighting a viable hybrid pathway. The study offers a practical route to more robust hand-keypoint inference in holistic pose estimation and provides open-source code to enable further optimization and integration.
Abstract
This paper addresses a critical flaw in MediaPipe Holistic's hand Region of Interest (ROI) prediction, which struggles with non-ideal hand orientations, affecting sign language recognition accuracy. We propose a data-driven approach to enhance ROI estimation, leveraging an enriched feature set including additional hand keypoints and the z-dimension. Our results demonstrate better estimates, with higher Intersection-over-Union compared to the current method. Our code and optimizations are available at https://github.com/sign-language-processing/mediapipe-hand-crop-fix.