milliMamba: Specular-Aware Human Pose Estimation via Dual mmWave Radar with Multi-Frame Mamba Fusion

TL;DR

<3-5 sentence high-level summary> milliMamba tackles the challenge of privacy-preserving mmWave radar 2D human pose estimation under specular-reflection sparsity by integrating spatio-temporal modeling in both encoding (Cross-View Fusion Mamba) and decoding (Multi-Pose STCA). It introduces 3D FFT-based heatmaps for efficient preprocessing, enabling long temporal context with linear complexity and dual-view fusion from orthogonal radars. Empirical results on TransHuPR and HuPR show substantial AP gains over baselines with competitive compute, establishing a new benchmark for radar-based HPE. The approach lays groundwork for robust multi-frame, dual-radar pose estimation and points to future multi-person and cross-environment applications.

Abstract

Millimeter-wave radar offers a privacy-preserving and lighting-invariant alternative to RGB sensors for Human Pose Estimation (HPE) task. However, the radar signals are often sparse due to specular reflection, making the extraction of robust features from radar signals highly challenging. To address this, we present milliMamba, a radar-based 2D human pose estimation framework that jointly models spatio-temporal dependencies across both the feature extraction and decoding stages. Specifically, given the high dimensionality of radar inputs, we adopt a Cross-View Fusion Mamba encoder to efficiently extract spatio-temporal features from longer sequences with linear complexity. A Spatio-Temporal-Cross Attention decoder then predicts joint coordinates across multiple frames. Together, this spatio-temporal modeling pipeline enables the model to leverage contextual cues from neighboring frames and joints to infer missing joints caused by specular reflections. To reinforce motion smoothness, we incorporate a velocity loss alongside the standard keypoint loss during training. Experiments on the TransHuPR and HuPR datasets demonstrate that our method achieves significant performance improvements, exceeding the baselines by 11.0 AP and 14.6 AP, respectively, while maintaining reasonable complexity. Code: https://github.com/NYCU-MAPL/milliMamba

Figures (5)

• Figure 1: Our milliMamba performs spatio-temporal modeling across both the feature extraction and decoding stages, addressing a key limitation of TransHuPR transhupr, which models these dependencies only partially. This is made possible by milliMamba’s ability to process a larger number of tokens with a comparable memory footprint, enabling richer temporal context and more accurate pose estimation.
• Figure 2: Overview of our milliMamba. The CVMamba encoder first extracts features from dual-view radar inputs. These features are then passed to the Multi-Pose STCA decoder, which progressively refines a set of keypoint queries to produce pose predictions.
• Figure 3: Our mamba scanning pattern. Only the forward direction is shown for clarity.
• Figure 4: Comparison of heatmap generation. (a) The traditional 4D approach radar_module applies separate FFTs for range, doppler, azimuth, and elevation after antenna grouping. (b) Our 3D pipeline performs a unified spatial FFT without grouping, yielding a compact representation. (c) Cost comparison between 4D and 3D heatmaps, showing 11× reduction in memory and 8.6× reduction in latency.
• Figure 5: Qualitative results on TransHuPR transhupr dataset.