Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB

TL;DR

This work tackles the ill-posed problem of estimating absolute 3D human pose from monocular RGB images by introducing depth-based Privileged Information during training. A three-backbone framework includes a depth-trained backbone, an RGB backbone, and a hallucination network that learns RGB features resembling depth features, enabling RGB-only inference with SPDH heatmaps. Through a two-stage training procedure and a fusion of RGB and hallucinated features, the method achieves improved accuracy on the Kinect MKV dataset, notably reducing $MPJPE$ and increasing $mAP$ at 10 cm thresholds, while maintaining real-time performance. The approach demonstrates the practicality of Privileged Information for 3D HPE and outlines avenues for using estimated depth from RGB to further generalize beyond dedicated depth sensors.

Abstract

Despite the recent advances in computer vision research, estimating the 3D human pose from single RGB images remains a challenging task, as multiple 3D poses can correspond to the same 2D projection on the image. In this context, depth data could help to disambiguate the 2D information by providing additional constraints about the distance between objects in the scene and the camera. Unfortunately, the acquisition of accurate depth data is limited to indoor spaces and usually is tied to specific depth technologies and devices, thus limiting generalization capabilities. In this paper, we propose a method able to leverage the benefits of depth information without compromising its broader applicability and adaptability in a predominantly RGB-camera-centric landscape. Our approach consists of a heatmap-based 3D pose estimator that, leveraging the paradigm of Privileged Information, is able to hallucinate depth information from the RGB frames given at inference time. More precisely, depth information is used exclusively during training by enforcing our RGB-based hallucination network to learn similar features to a backbone pre-trained only on depth data. This approach proves to be effective even when dealing with limited and small datasets. Experimental results reveal that the paradigm of Privileged Information significantly enhances the model's performance, enabling efficient extraction of depth information by using only RGB images.

Figures (7)

• Figure 1: Overview of the proposed 3D human pose estimator (3D HPE) approach. The method is based on the paradigm of Privileged Information vapnik2009new, consisting of providing additional depth data during the training phase. At inference time, the system works only with RGB images.
• Figure 2: The general overview of the proposed method. Adopting the Privileged Learning vapnik2009learning paradigm, RGB and additional depth data are provided during the training stage. In this manner, through a specific loss, we force the hallucination network, to extract features resembling the ones learned by a model pretrained on depth images. These features are concatenated to the visual RGB features and then given as input to the pose estimation branch based on the SPDH simoni2022semi representation. Finally, the 3D human pose is predicted in world coordinates.
• Figure 3: Visual representation of the $F_{depth}$ model (SmallHRNet sun2019high backbone) that predicts the 3D human pose exploiting the SPDH representation. This is the first step of the whole training procedure based on the Privileged Information paradigm.
• Figure 4: A detailed visualization of the pose estimation branch that leverages the fusion of multi-resolution features to improve the accuracy of the predicted SPDH heatmaps. The input is the concatenation of the feature maps extracted through the hallucination network and the RGB-based one (see Fig. \ref{['fig:overview']}).
• Figure 5: Some visual examples of the Kinect Human Pose Dataset zimmermann20183d in terms of RGB (first row) and depth images (second row). As shown, a variety of body poses, environments, and subject distances, characterize the dataset. Depth maps are depicted in 8-bit format only for visualization.