SkeFi: Cross-Modal Knowledge Transfer for Wireless Skeleton-Based Action Recognition

TL;DR

SkeFi tackles privacy-preserving skeleton-based action recognition using non-intrusive wireless sensors (LiDAR/mmWave) by transferring knowledge from RGB-based skeleton data. It introduces a temporal correlation-augmented graph convolution (TC-AGCN) and additive ESP-MST multi-scale temporal modeling to cope with frame loss and noise, enabling robust cross-modal transfer from RGB (Kinetics-Skeleton) to MM-Fi (RGB, LiDAR, mmWave). A two-stage, layer-freezing transfer strategy preserves modality-agnostic structure while adapting to target-domain noise, achieving strong performance across wireless modalities and demonstrating data-efficient learning. The approach offers practical benefits for smart homes and hospitals by reducing privacy concerns and reliance on lighting conditions, with the MM-Fi+Kinetics-Skeleton transfer showing significant improvements over training-from-scratch baselines.

Abstract

Skeleton-based action recognition leverages human pose keypoints to categorize human actions, which shows superior generalization and interoperability compared to regular end-to-end action recognition. Existing solutions use RGB cameras to annotate skeletal keypoints, but their performance declines in dark environments and raises privacy concerns, limiting their use in smart homes and hospitals. This paper explores non-invasive wireless sensors, i.e., LiDAR and mmWave, to mitigate these challenges as a feasible alternative. Two problems are addressed: (1) insufficient data on wireless sensor modality to train an accurate skeleton estimation model, and (2) skeletal keypoints derived from wireless sensors are noisier than RGB, causing great difficulties for subsequent action recognition models. Our work, SkeFi, overcomes these gaps through a novel cross-modal knowledge transfer method acquired from the data-rich RGB modality. We propose the enhanced Temporal Correlation Adaptive Graph Convolution (TC-AGC) with frame interactive enhancement to overcome the noise from missing or inconsecutive frames. Additionally, our research underscores the effectiveness of enhancing multiscale temporal modeling through dual temporal convolution. By integrating TC-AGC with temporal modeling for cross-modal transfer, our framework can extract accurate poses and actions from noisy wireless sensors. Experiments demonstrate that SkeFi realizes state-of-the-art performances on mmWave and LiDAR. The code is available at https://github.com/Huang0035/Skefi.

Figures (7)

• Figure 1: SkeFi uses non-invasive wireless sensors to recognize human actions through skeletal movements. It deals with the lack of wireless sensor data by cross-modal transfer learning.
• Figure 2: (a). SkeFi, a transfer learning framework. It trains the model using RGB modality in the source domain and transfers prior knowledge to the target domain by freezing parameters. Green and blue denote trainable and frozen parameters, respectively. (b). TC-AGC. We propose a temporal correlation module, illustrated in the blue box, to boost connectivity in frame sequences by facilitating better interaction between missing and normal frames. $\bigoplus$ denotes the element-wise sum. $\bigotimes$ denotes the matrix multiplication.
• Figure 3: The left sketch shows the joint label of the Kinetics-Skeleton dataset, and the right sketch shows the joint label extracted from the MM-Fi dataset.
• Figure 4: Visualization of human pose estimation using three modalities. A red dot is keypoint we added.
• Figure 5: The training process of AGCN in three modalities, where the dotted line represents training from scratch, and the solid line represents transfer learning.