RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequence

TL;DR

RadarLLM is presented, the first framework that leverages large language models (LLMs) for human motion understanding from radar signals with state-of-the-art performance, enabling robust and interpretable motion understanding under privacy and visibility constraints, even in adverse environments.

Abstract

Millimeter-wave radar offers a privacy-preserving and environment-robust alternative to vision-based sensing, enabling human motion analysis in challenging conditions such as low light, occlusions, rain, or smoke. However, its sparse point clouds pose significant challenges for semantic understanding. We present RadarLLM, the first framework that leverages large language models (LLMs) for human motion understanding from radar signals. RadarLLM introduces two key innovations: (1) a motion-guided radar tokenizer based on our Aggregate VQ-VAE architecture, integrating deformable body templates and masked trajectory modeling to convert spatial-temporal radar sequences into compact semantic tokens; and (2) a radar-aware language model that establishes cross-modal alignment between radar and text in a shared embedding space. To overcome the scarcity of paired radar-text data, we generate a realistic radar-text dataset from motion-text datasets with a physics-aware synthesis pipeline. Extensive experiments on both synthetic and real-world benchmarks show that RadarLLM achieves state-of-the-art performance, enabling robust and interpretable motion understanding under privacy and visibility constraints, even in adverse environments. This paper has been accepted for presentation at AAAI 2026. This is an extended version with supplementary materials.

Figures (9)

• Figure 1: We propose RadarLLM, a LLM-based radar-text human motion understanding framework over traditional action label-based motion recognition in providing descriptive natural language insights, recognizing unconventional motions beyond predefined categories, and operating robustly in adverse conditions (e.g., poor lighting, occlusion, rain, and smoke).
• Figure 2: Virtual radar-text data generation pipeline. The Radar-Text dataset is constructed by simulating radar reflections from SMPL-X sequences using ray tracing and signal processing, based on existing motion-text datasets.
• Figure 3: The overview of RadarLLM. We first encode radar point clouds into discrete tokens via a Motion-guided Radar Tokenizer. The Radar-aware Language Model then aligns these tokens with textual representations in a shared embedding space through joint optimization of unsupervised token reconstruction and supervised bidirectional radar-text translation.
• Figure 4: Architecture and training pipeline of motion-guided radar tokenizer. The Motion-guided Radar Tokenizer, built upon our Aggregate VQ-VAE architecture, compresses radar point cloud sequences into discrete semantic tokens through point cloud sequence reconstruction and motion embedding learning.
• Figure 5: Visualization of predicted textual descriptions alongside corresponding motion sequences and radar point clouds. The left three columns demonstrate results on real-world normal environment data, while the right two columns showcase predictions on synthesized virtual data.