In-Bed Human Pose Estimation from Unseen and Privacy-Preserving Image Domains

TL;DR

This work tackles privacy-aware in-bed pose estimation where RGB data cannot be used in clinical settings. It introduces a multi-modal conditional variational autoencoder (MC-VAE) that learns cross-modality feature distributions and reconstructs the missing RGB features from LWIR during inference, enabling single-modality operation. Built on HRNet, the approach demonstrates strong pose-estimation performance on the SLP dataset, outperforming baselines that rely on both modalities or on missing-modality schemes, and shows robust generalization to unseen hospital environments. The results highlight the practical impact of privacy-preserving, self-supervised learning for inpatient monitoring with limited labeled data and variable sensor availability.

Abstract

Medical applications have benefited greatly from the rapid advancement in computer vision. Considering patient monitoring in particular, in-bed human posture estimation offers important health-related metrics with potential value in medical condition assessments. Despite great progress in this domain, it remains challenging due to substantial ambiguity during occlusions, and the lack of large corpora of manually labeled data for model training, particularly with domains such as thermal infrared imaging which are privacy-preserving, and thus of great interest. Motivated by the effectiveness of self-supervised methods in learning features directly from data, we propose a multi-modal conditional variational autoencoder (MC-VAE) capable of reconstructing features from missing modalities seen during training. This approach is used with HRNet to enable single modality inference for in-bed pose estimation. Through extensive evaluations, we demonstrate that body positions can be effectively recognized from the available modality, achieving on par results with baseline models that are highly dependent on having access to multiple modes at inference time. The proposed framework supports future research towards self-supervised learning that generates a robust model from a single source, and expects it to generalize over many unknown distributions in clinical environments.

Figures (4)

• Figure 1: Overview of the proposed self-supervised framework. Offline training: a feature reconstruction module is adopted to learn the distribution of a modality which is absent at inference time through a variational autoencoder. During training we have access to both modalities, RGB and LWIR in the current setup. Online inference: the system only requires information from the available modality (LWIR) as the features of the missing modality can be reconstructed.
• Figure 2: Comparison between baseline models and our self-supervised approach. Models (\ref{['fig:fine-tuning']}), (\ref{['fig:feature-fusion']}) and (\ref{['fig:rdf']}) are based on existing methods. (\ref{['fig:fine-tuning']}) is trained on the secondary modality (RGB) then we froze the stem layer, fine-tuning the HRNet and final decision layer on the first modality (LWIR). (\ref{['fig:feature-fusion']}) images from both modalities are required as input, a late-layer fusion module based on IAFF dai2021attentional fuses the late features from parallel HRNet model. (\ref{['fig:rdf']}) To simulate the missing modality scenario at inference time, RDF karanam2020towards randomly sets the RGB image to 0 during training, during inference the RGB image is always 0 to represent the missing modality. When the RGB image input is 0, the model skips the fusion and uses features from IR streams directly (dotted line). (\ref{['fig:ours']}) Our model leverages features from both modalities during training and reconstructs the missing modality conditioned on the available modality during inference.
• Figure 3: An illustration of the training and testing scheme. (a) During training, both modalities are available, the model first extracts features $\mathcal{X}_1$ and $\mathcal{X}_2$ from each modality (shown in pink). Then the conditional variational autoencoder learns the joint representation of the features from both modalities, and reconstructs feature $\hat{\mathcal{X}_2}$ conditioned on $\mathcal{X}_1$. The reconstructed feature $\hat{\mathcal{X}_2}$ is later trained in parallel with $\mathcal{X}_1$, then fused together via the Fusion module. The final fused features are used to predict joint locations. (b) During testing, modality 2 is unavailable, but the conditional variational autoencoder can reconstruct $\hat{\mathcal{X}_2}$ conditioned on $\mathcal{X}_1$ without input $\mathcal{X}_2$.
• Figure 4: Qualitative results of human pose estimation on covered and uncovered LWIR images. RGB images are only shown to aid results for visualisation. Inference from (a) uncovered, (b-d) covered.