BlanketGen2-Fit3D: Synthetic Blanket Augmentation Towards Improving Real-World In-Bed Blanket Occluded Human Pose Estimation

TL;DR

This work tackles the challenge of 2D human pose estimation from RGB images when blankets occlude the body in bed, a setting with scarce labeled data. It introduces BlanketGen2-Fit3D, a two-step cloth-simulation and rendering pipeline that augments the Fit3D dataset with photorealistic synthetic blankets, and uses it to finetune ViTPose-B. The mixed training on Fit3D and BlanketGen2-Fit3D improves occlusion robustness, achieving higher PCK and lower NME on synthetic blankets and better transfer to real blankets in the SLP dataset. The results demonstrate the practical potential of synthetic blanket augmentation to bridge domain gaps in in-bed occluded HPE, and the authors provide the dataset and code for public use.

Abstract

Human Pose Estimation (HPE) from monocular RGB images is crucial for clinical in-bed skeleton-based action recognition, however, it poses unique challenges for HPE models due to the frequent presence of blankets occluding the person, while labeled HPE data in this scenario is scarce. To address this we introduce BlanketGen2-Fit3D (BG2-Fit3D), an augmentation of Fit3D dataset that contains 1,217,312 frames with synthetic photo-realistic blankets. To generate it we used BlanketGen2, our new and improved version of our BlanketGen pipeline that simulates synthetic blankets using ground-truth Skinned Multi-Person Linear model (SMPL) meshes and then renders them as transparent images that can be layered on top of the original frames. This dataset was used in combination with the original Fit3D to finetune the ViTPose-B HPE model, to evaluate synthetic blanket augmentation effectiveness. The trained models were further evaluated on a real-world blanket occluded in-bed HPE dataset (SLP dataset). Comparing architectures trained on only Fit3D with the ones trained with our synthetic blanket augmentation the later improved pose estimation performance on BG2-Fit3D, the synthetic blanket occluded dataset significantly to (0.977 Percentage of Correct Keypoints (PCK), 0.149 Normalized Mean Error (NME)) with an absolute 4.4% PCK increase. Furthermore, the test results on SLP demonstrated the utility of synthetic data augmentation by improving performance by an absolute 2.3% PCK, on real-world images with the poses occluded by real blankets. These results show synthetic blanket augmentation has the potential to improve in-bed blanket occluded HPE from RGB images. The dataset as well as the code will be made available to the public.

Figures (7)

• Figure 1: On the left: different stages of the BlanketGen2 pipeline with the improvements introduced in this paper. On the right: pose predictions on a frame from BlanketGen2-Fit3D and another from SLP with two different fine-tuned ViTPose-B models vitpose, one fine-tuned on Fit3D (FT-Fit3D) and the other on a mixed dataset of Fit3D and BlanketGen2-Fit3D (FT-Mixed).
• Figure 2: The scene from BlanketGen2-Fit3D, the four big rectangles are area lights set up to mimic the light reflecting off the walls and ceiling in the room; the rectangular cuboid behind the person serves the purpose of a bed for the physics of the blanket and is completely excluded from the rendering; the pyramids are representing the cameras.
• Figure 3: Comparison of the BlanketGen and BlanketGen2 data processing pipelines, highlighting the enhanced efficiency and versatility of the new setup (b). The BlanketGen2 pipeline also introduces the capability for unrestricted sharing of generated blankets, independent of the source dataset, and supports flexible retexturing options.
• Figure 4: The blanket texture used in the old BlanketGen pipeline on the left, and on the right the new, realistic texture used for BlanketGen2-Fit3D.
• Figure 5: (a) A frame from SLP, the black and white circles are the ground-truth joints and the green circles are the joint predictions from the model with the VitPose-B model with the head fine-tuned on Fit3D (FT-Fit3D). Note how the hip joints are offset: in the SLP format, they are annotated on the axis of rotation of the leg, while in the Fit3D format, they are annotated at the outside of the hip. (b): A zoomed-in frame from Fit3D with the ground-truth joints in green.