Advancing Radar Hand Gesture Recognition: A Hybrid Spectrum Synthetic Framework Merging Simulation with Neural Networks

TL;DR

This work tackles the limited radar data problem in mmWave hand gesture recognition by introducing a hybrid spectrum synthesis framework that marries a physics-based cylinder-mesh hand model with an adaptive neural reweighting network, RadarWeightNet. Trained on paired radar and vision data, the system produces high-fidelity synthetic spectra that, when used for data augmentation, dramatically improves few-shot HGR performance and enables transfer from external vision datasets. Key contributions include a novel per-reflection-point weighting mechanism, extensive OOD and few-shot evaluations, and evidence that the approach preserves physical interpretability while delivering data-driven adaptability. The results demonstrate meaningful improvements in spectrum fidelity (SSIM up to 63.0) and gesture recognition accuracy in data-scarce settings, validating the method as a practical augmentation strategy for privacy-preserving radar-based gesture systems.

Abstract

Millimeter wave (mmWave) radar sensors play a vital role in hand gesture recognition (HGR) by detecting subtle motions while preserving user privacy. However, the limited scale of radar datasets hinders the performance. Existing synthetic data generation methods fall short in two key areas. On the one hand, modeling-based approaches fail to accurately simulate the wave propagation and reflection at the hand-gesture level, facing unique complexities such as diffraction and occlusion. On the other hand, generative model-based methods are hard to converge while radar data is limited, lacking interpretability, and sometimes fail to produce kinematically plausible results. To overcome these limitations, we propose a novel hybrid spectrum synthetic framework leveraging visual hand gesture data. It combines a cylinder mesh-based hand reflection model with a small-scale neural network called RadarWeightNet, which focuses on assigning weights to simulated signals. Our framework addresses two key challenges: achieving accurate simulation of complex hand geometry and bridging the simulation-to-real gap in a data-driven manner while preserving interpretability, which balances physical accuracy with machine learning adaptability. We tested our framework under extreme scenarios where radar data is scarce. The results demonstrate the effectiveness of our hybrid framework, achieving up to 63% SSIM in synthetic performance and up to 30% improvement in classification performance in few-shot learning.

Figures (9)

• Figure 1: The system overview of our hybrid synthetic framework and how it enhances the HGR accuracy by easing the dataset scale limitation. The training process of the synthetic framework (green path) uses paired mmWave radar data and 3D hand coordinates to optimize RadarWeightNet, which reweights modeled radar signals from a cylinder mesh-based hand model. The inference phase (red path) generates synthetic radar spectrums from external vision modality and then expands the spectrum dataset, improving the classification performances in radar-based hand gesture recognition.
• Figure 2: Radar signal processing pipeline.
• Figure 3: Hand modeling pipeline overview. A camera captures hand gestures and estimates 3D joint coordinates, which are used to construct a cylinder-based 3D mesh model of the hand. This model accurately represents hand geometry and occlusion, and is transformed into the radar coordinate system. The resulting detailed hand model enables the simulation of radar reflections from multiple scattering points, allowing for realistic radar signal synthesis that accounts for complex hand structures and movements.
• Figure 4: Detailed architecture of RadarWeightNet.
• Figure 5: Hardware setup of data aquirsation