MoCap2Radar: A Spatiotemporal Transformer for Synthesizing Micro-Doppler Radar Signatures from Motion Capture
Kevin Chen, Kenneth W. Parker, Anish Arora
TL;DR
This work tackles radar data scarcity by learning a pure data-driven mapping from Motion-Capture sequences to micro-Doppler radar spectrograms. It introduces a spatiotemporal transformer with separate spatial and temporal blocks and a cross-attention fusion to produce Doppler spectra from windowed MoCap inputs, trained end-to-end with MSE on STFT-derived targets. Experiments on real MoCap-radar data demonstrate convergence, generalization to novel walking patterns, and the importance of jointly modeling spatial and temporal dynamics for accurate synthesis. The results indicate potential for MoCap-based radar data augmentation and efficient data generation for edge IoT radars, with implications for transfer learning across sensing modalities.
Abstract
We present a pure machine learning process for synthesizing radar spectrograms from Motion-Capture (MoCap) data. We formulate MoCap-to-spectrogram translation as a windowed sequence-to-sequence task using a transformer-based model that jointly captures spatial relations among MoCap markers and temporal dynamics across frames. Real-world experiments show that the proposed approach produces visually and quantitatively plausible doppler radar spectrograms and achieves good generalizability. Ablation experiments show that the learned model includes both the ability to convert multi-part motion into doppler signatures and an understanding of the spatial relations between different parts of the human body. The result is an interesting example of using transformers for time-series signal processing. It is especially applicable to edge computing and Internet of Things (IoT) radars. It also suggests the ability to augment scarce radar datasets using more abundant MoCap data for training higher-level applications. Finally, it requires far less computation than physics-based methods for generating radar data.