SNN-Driven Multimodal Human Action Recognition via Sparse Spatial-Temporal Data Fusion

TL;DR

This work targets the high energy and memory costs of RGB-skeleton fusion in human action recognition by introducing an SNN-based multimodal framework that fuses event camera data with skeletons. It couples a Skeleton SGN and an Event Spiking Mamba backbone, augmented by Sparse Semantic Extractor and Spiking Cross Mamba for cross-modal fusion, and a two-stage Discretized Information Bottleneck to compress features while preserving task-relevant semantics. A novel event-skeleton data construction pipeline enables thorough evaluation, and extensive experiments show state-of-the-art accuracy with substantially lower energy consumption compared with ANN baselines. The approach advances energy-efficient neuromorphic computing for practical HAR and suggests future directions like MT-UI, real-world event datasets, and deployment on neuromorphic hardware.

Abstract

Multimodal human action recognition based on RGB and skeleton data fusion, while effective, is constrained by significant limitations such as high computational complexity, excessive memory consumption, and substantial energy demands, particularly when implemented with Artificial Neural Networks (ANN). These limitations restrict its applicability in resource-constrained scenarios. To address these challenges, we propose a novel Spiking Neural Network (SNN)-driven framework for multimodal human action recognition, utilizing event camera and skeleton data. Our framework is centered on two key innovations: (1) a novel multimodal SNN architecture that employs distinct backbone networks for each modality-an SNN-based Mamba for event camera data and a Spiking Graph Convolutional Network (SGN) for skeleton data-combined with a spiking semantic extraction module to capture deep semantic representations; and (2) a pioneering SNN-based discretized information bottleneck mechanism for modality fusion, which effectively balances the preservation of modality-specific semantics with efficient information compression. To validate our approach, we propose a novel method for constructing a multimodal dataset that integrates event camera and skeleton data, enabling comprehensive evaluation. Extensive experiments demonstrate that our method achieves superior performance in both recognition accuracy and energy efficiency, offering a promising solution for practical applications.

Figures (12)

• Figure 1: Comparison of vanilla ANN-based RGB-Skeleton fusion (a) and our proposed SNN-based event-skeleton fusion (b) for human action recognition.
• Figure 2: Overview of the proposed SNN-driven multimodal action recognition framework. The left panel shows the full pipeline; right subfigures detail core modules. (a) SGN and (b) Spiking Mamba extract features from skeleton and event data, respectively. (c) SSE enhances modality-specific semantics via structural self-similarity. (d) GSA, within SSE, improves global spiking feature alignment. (e) SCM enables cross-modal interaction through selective and state-space paths. The bottom-left illustrates the two-stage DIB module, which compresses fused features before classification.
• Figure 3: Feature Space Visualization. The grayscale intensity represents the activation level of the spiking neurons at each spatiotemporal point.
• Figure 4: t-SNE visualization of feature distributions. Different colors indicate different actions.
• Figure 5: Hyperparameter ($\alpha,\lambda_1,\lambda_2$) analysis of DIB.

Theorems & Definitions (13)

• Proposition 1: Minimal Gaussian--Gaussian KL under a fixed spike probability
• proof : Proof sketch
• Proposition 2: Hard thresholds are not pathwise differentiable
• proof : Proof sketch
• Lemma 1: Cross-entropy gradients explode near 0/1 and vanish at agreement
• Theorem 1: Condition number scales as the inverse sparsity
• proof : Proof sketch
• Proposition 3: Discrete KL upper-bounds mutual information and controls spikes
• Lemma 2: Cosine similarity as a stable MI surrogate under sparsity
• Lemma 3: Spike-domain sampling preserves first moments and stabilizes gradients