mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction

TL;DR

This work addresses privacy-preserving human body reconstruction from sparse mmWave radar data by introducing mmDEAR, a two-stage framework. Stage 1 densifies mmWave point clouds through a temporal feature extractor and cascaded completion guided by 2D masks during training, producing an enhanced representation without image data during inference. Stage 2 performs 2D-3D aware reconstruction by fusing raw and enhanced clouds to regress SMPL-X parameters, using a P4Conv-based motion encoder and transformer-based 2D-3D fusion. Experiments on mmBody and MRI datasets show significant improvements over state-of-the-art methods in MPJPE, MPVPE, and related metrics, and demonstrate that the enhancement module can boost existing methods while preserving inference privacy. The approach offers practical impact for robust, privacy-conscious human reconstruction in varied environments and devices.

Abstract

Millimeter-wave (mmWave) radar offers robust sensing capabilities in diverse environments, making it a highly promising solution for human body reconstruction due to its privacy-friendly and non-intrusive nature. However, the significant sparsity of mmWave point clouds limits the estimation accuracy. To overcome this challenge, we propose a two-stage deep learning framework that enhances mmWave point clouds and improves human body reconstruction accuracy. Our method includes a mmWave point cloud enhancement module that densifies the raw data by leveraging temporal features and a multi-stage completion network, followed by a 2D-3D fusion module that extracts both 2D and 3D motion features to refine SMPL parameters. The mmWave point cloud enhancement module learns the detailed shape and posture information from 2D human masks in single-view images. However, image-based supervision is involved only during the training phase, and the inference relies solely on sparse point clouds to maintain privacy. Experiments on multiple datasets demonstrate that our approach outperforms state-of-the-art methods, with the enhanced point clouds further improving performance when integrated into existing models.

Figures (4)

• Figure 1: Our proposed mmDEAR method first enhances the sparse point cloud to increase its density and capture detailed body posture and shape information, improving the accuracy of the final body reconstruction. The enhancement module is guided by a 2D human mask extracted from a single-view image during training, ensuring that the inference process is both image-free and privacy-preserving.
• Figure 2: The framework of our proposed mmDEAR, comprising image-aided point cloud enhancement module and 2D-3D aware human body reconstruction module. These two modules are trained separately, with the trained enhancement module serving as a pre-processing step for the reconstruction module during both the training and inference processes.
• Figure 3: Qualitative Results for Point Cloud Enhancement and Reconstruction
• Figure 4: Ablation Study Results on Point Cloud Enhancement and 2D Joint Supervision