Privacy-Preserving In-Bed Pose Monitoring: A Fusion and Reconstruction Study

TL;DR

This work tackles in-bed pose estimation under bedding occlusions and privacy constraints by leveraging privacy-preserving modalities (depth, LWIR, and pressure) and a cGAN-based pathway to reconstruct visible data when unavailable. It develops multiple HRNet-based fusion strategies—addition, concatenation, learned modal weights, and an end-to-end approach—alongside a Visible-LWIR reconstruction pipeline to enable fusion without real visible data. Experiments on the SLP dataset across DanaLab and SimLab settings show end-to-end fusion generally yields the strongest performance and can generalize across environments, with synthesized visible images achieving around $PCKh@0.5$ = $90.2\%$. This privacy-conscious framework broadens the practicality of hospital in-bed monitoring by reducing privacy risks while maintaining high pose-estimation accuracy.

Abstract

Recently, in-bed human pose estimation has attracted the interest of researchers due to its relevance to a wide range of healthcare applications. Compared to the general problem of human pose estimation, in-bed pose estimation has several inherent challenges, the most prominent being frequent and severe occlusions caused by bedding. In this paper we explore the effective use of images from multiple non-visual and privacy-preserving modalities such as depth, long-wave infrared (LWIR) and pressure maps for the task of in-bed pose estimation in two settings. First, we explore the effective fusion of information from different imaging modalities for better pose estimation. Secondly, we propose a framework that can estimate in-bed pose estimation when visible images are unavailable, and demonstrate the applicability of fusion methods to scenarios where only LWIR images are available. We analyze and demonstrate the effect of fusing features from multiple modalities. For this purpose, we consider four different techniques: 1) Addition, 2) Concatenation, 3) Fusion via learned modal weights, and 4) End-to-end fully trainable approach; with a state-of-the-art pose estimation model. We also evaluate the effect of reconstructing a data-rich modality (i.e., visible modality) from a privacy-preserving modality with data scarcity (i.e., long-wavelength infrared) for in-bed human pose estimation. For reconstruction, we use a conditional generative adversarial network. We conduct ablative studies across different design decisions of our framework. This includes selecting features with different levels of granularity, using different fusion techniques, and varying model parameters. Through extensive evaluations, we demonstrate that our method produces on par or better results compared to the state-of-the-art.

Figures (12)

• Figure 1: The HRNet architecture consists of 4 stages. In each stage, branches are arranged to process high-to-low resolution feature maps. At the end of each stage except the final stage, the information is exchanged between branches of the next stage. The input to the model is an $N$$\times$ 256 $\times$ 256 image, where $N$ is the number of image channels and varies with the modality (for the visible modality $N$=3, for other modalities $N$=1). The generated output contains 14 64 $\times$ 64 heatmaps, corresponding to 14 predicted key points. Recreated from 8953615.
• Figure 2: Intermediate fusion using addition at stage two of HR-NET with LWIR and depth modalities.
• Figure 3: Intermediate fusion using concatenation at stage two of HR-NET with LWIR and depth modalities.
• Figure 4: Fusion with learned channel-wise mode weights, applied to stage two of HR-Net with LWIR and depth modalities.
• Figure 5: End-to-end fully trainable approach applied to stage two of HR-NET with LWIR and depth modalities.