RSB-Pose: Robust Short-Baseline Binocular 3D Human Pose Estimation with Occlusion Handling

TL;DR

RSB-Pose tackles robust 3D human pose estimation with short-baseline binoculars, addressing depth robustness under 2D errors and occlusions. The method introduces Stereo Co-Keypoints Estimation (SCE) with a Stereo Volume Feature (SVF) to jointly regress binocular 2D keypoints with guaranteed correspondences across disparities. Additionally, a Pre-trained Pose Transformer (PPT) refines 3D poses by modeling pose coherence among joints, trained via a self-supervised masked-joint recovery task. Evaluations on MHAD and H36M demonstrate improved 3D accuracy and occlusion handling, particularly in short-baseline settings, with ablations confirming the contributions of SCE and PPT. Overall, RSB-Pose offers a portable, occlusion-robust solution for binocular HPE and highlights the potential for disparity-driven cross-view reasoning.

Abstract

In the domain of 3D Human Pose Estimation, which finds widespread daily applications, the requirement for convenient acquisition equipment continues to grow. To satisfy this demand, we set our sights on a short-baseline binocular setting that offers both portability and a geometric measurement property that radically mitigates depth ambiguity. However, as the binocular baseline shortens, two serious challenges emerge: first, the robustness of 3D reconstruction against 2D errors deteriorates; and second, occlusion reoccurs due to the limited visual differences between two views. To address the first challenge, we propose the Stereo Co-Keypoints Estimation module to improve the view consistency of 2D keypoints and enhance the 3D robustness. In this module, the disparity is utilized to represent the correspondence of binocular 2D points and the Stereo Volume Feature is introduced to contain binocular features across different disparities. Through the regression of SVF, two-view 2D keypoints are simultaneously estimated in a collaborative way which restricts their view consistency. Furthermore, to deal with occlusions, a Pre-trained Pose Transformer module is introduced. Through this module, 3D poses are refined by perceiving pose coherence, a representation of joint correlations. This perception is injected by the Pose Transformer network and learned through a pre-training task that recovers iterative masked joints. Comprehensive experiments carried out on H36M and MHAD datasets, complemented by visualizations, validate the effectiveness of our approach in the short-baseline binocular 3D Human Pose Estimation and occlusion handling.

Figures (8)

• Figure 1: Two main challenges of short-baseline binocular 3D human pose estimation: A. 3D reconstruction robustness against 2D keypoint errors deteriorates; B. occlusion re-emerges in both views. In A, the yellow and green intersection zones show the horizontal tangent plane of uncertainty region under different baselines respectively. In B, the green boxes indicate the left image, while yellow boxes represent the right one. The white circles indicate the occluded point, and the yellow ones are visible.
• Figure 2: The framework of RSB-Pose. The binocular images are firstly encoded by a 2D backbone and then processed through three main steps: I. Stereo Co-Keypoints Generation: Two-view features are concatenated in the Stereo Volume Feature (SVF), facilitating the simultaneous regression of 2D binocular keypoints and ensuring their view consistency; II. 3D Pose Initialization: Triangulation is utilized to reconstruct the initial 3D pose; III. 3D Pose PPT Refinement: Pose coherence is perceived by the Pose Transformer through pre-training and then injected into the refined 3D pose.
• Figure 3: The framework of Stereo Co-Keypoints Estimation module: I. Attention Mask Generation, to focus initial features on the huaman body of interest; II. Stereo Volume Feature Generation, to consider both binocular views simultaneously and form as a 4D feature volume; III. 2D Binocular Dismantling, to solve binocular 2D keypoints from co-keypoints regressed from SVF.
• Figure 4: Illustration of Pre-trained Pose Transformer: A. Pre-Training Strategy, B. End-to-End Training within the framework, C. Pose Transformer Structure. Firstly, the Pose Transformer undergoes a pre-training stage with a self-supervised task, involving iterative recovery of masked poses. Subsequently, during the whole framework end-to-end training, the Pre-trained Pose Transformer is reloaded and receives initial predicted 3D poses as input.
• Figure 5: Qualitative comparison with SOTA methods. The images are all captured from the left view. The number under each pose corresponds to the MPJPE_re result. The gray skeleton represents the groundtruth, while the black skeleton represents the estimated pose. In the black skeleton, right joints are marked in red, and left joints are marked in blue. The left half shows results on the H36M dataset and the right half is on the MHAD dataset. Yellow arrows indicate parts of significant improvement in our method compared to SOTA methods.