Signal-SGN++: Topology-Enhanced Time-Frequency Spiking Graph Network for Skeleton-Based Action Recognition

TL;DR

Signal-SGN++ tackles skeleton-based action recognition by integrating topology-aware spiking graphs with time-frequency analysis to achieve energy-efficient yet accurate performance. The method couples a 1D Spiking Graph Convolution backbone with Frequency Spiking Convolution, augmented by a Topology-Shift Self-Attention and a Multi-Scale Wavelet Transform Fusion that preserves topology during spectral fusion. Legendre-polynomial-based wavelet filters and topology-guided fusion (TATF) enable robust multi-resolution time-frequency modeling, while a topology-aware MWTF branch enriches representations. Experiments on NTU RGB+D and NTU 120 demonstrate competitive accuracy with substantial energy savings, highlighting the viability of neuromorphic approaches for large-scale skeleton action recognition.

Abstract

Graph Convolutional Networks (GCNs) demonstrate strong capability in modeling skeletal topology for action recognition, yet their dense floating-point computations incur high energy costs. Spiking Neural Networks (SNNs), characterized by event-driven and sparse activation, offer energy efficiency but remain limited in capturing coupled temporal-frequency and topological dependencies of human motion. To bridge this gap, this article proposes Signal-SGN++, a topology-aware spiking graph framework that integrates structural adaptivity with time-frequency spiking dynamics. The network employs a backbone composed of 1D Spiking Graph Convolution (1D-SGC) and Frequency Spiking Convolution (FSC) for joint spatiotemporal and spectral feature extraction. Within this backbone, a Topology-Shift Self-Attention (TSSA) mechanism is embedded to adaptively route attention across learned skeletal topologies, enhancing graph-level sensitivity without increasing computational complexity. Moreover, an auxiliary Multi-Scale Wavelet Transform Fusion (MWTF) branch decomposes spiking features into multi-resolution temporal-frequency representations, wherein a Topology-Aware Time-Frequency Fusion (TATF) unit incorporates structural priors to preserve topology-consistent spectral fusion. Comprehensive experiments on large-scale benchmarks validate that Signal-SGN++ achieves superior accuracy-efficiency trade-offs, outperforming existing SNN-based methods and achieving competitive results against state-of-the-art GCNs under substantially reduced energy consumption.

Figures (10)

• Figure 1: Motivating Perspective of Signal-SGN++. Skeleton data is encoded through a spiking neural network, transforming it into a sparse binary representation that is interpreted as a multi-dimensional discrete signal. This approach enables the application of time-frequency analysis, focusing on the extraction of spectral-temporal patterns, which is a critical component for enhancing action recognition performance in Signal-SGN++.
• Figure 2: Signal-SGN++ Architecture. The model consists of a series of stacked modules: 1D-Spiking Graph Convolution (1D-SGC), followed by Topology-Shift Self-Attention (TSSA), then Frequency Spiking Convolution (FSC), and finally the Multi-Scale Wavelet Transform Fusion (MWTF) for feature fusion and classification.
• Figure 3: Multi-Scale Wavelet Transform and Topological Fusion. (a) The overall framework of the MWTF module, which integrates multi-scale wavelet transforms and topological fusion. (b) The complete process from DWT to TATF, including the alignment of time dimensions, decomposition into high-frequency and low-frequency components, and topological neighborhood fusion.
• Figure 4: Layer-wise and accumulated topology score matrices. Off-diagonal mass increases with depth, revealing emergent nonlocal couplings and action-specific coordination.
• Figure 5: Learned joint-wise windows $w_V$ across layers (L1--L4). Shallow layers emphasize distal effectors; deeper layers smooth around the trunk, enlarging the effective receptive field.