Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning

TL;DR

The paper addresses robust cross-modal fusion in spiking neural networks (SNNs) for audio-visual tasks. It introduces semantic-alignment cross-modal residual learning (S-CMRL), a Transformer-based multimodal SNN featuring a cross-modal complementary spatiotemporal spiking attention (CCSSA) and a semantic alignment optimization (SAO) module. CCSSA treats cross-modal information as residuals to preserve unimodal semantics, while SAO aligns cross-modal residuals in a shared semantic space using a loss $\mathcal{L}_{sao}$, yielding $\mathcal{L}=\mathcal{L}_{ce}+\mathcal{L}_{sao}$. Experiments on CREMA-D, UrbanSound8K-AV, and MNISTDVS-NTIDIGITS show state-of-the-art accuracy and strong noise robustness, demonstrating the practical potential of semantic-aligned multimodal SNNs.

Abstract

Humans interpret and perceive the world by integrating sensory information from multiple modalities, such as vision and hearing. Spiking Neural Networks (SNNs), as brain-inspired computational models, exhibit unique advantages in emulating the brain's information processing mechanisms. However, existing SNN models primarily focus on unimodal processing and lack efficient cross-modal information fusion, thereby limiting their effectiveness in real-world multimodal scenarios. To address this challenge, we propose a semantic-alignment cross-modal residual learning (S-CMRL) framework, a Transformer-based multimodal SNN architecture designed for effective audio-visual integration. S-CMRL leverages a spatiotemporal spiking attention mechanism to extract complementary features across modalities, and incorporates a cross-modal residual learning strategy to enhance feature integration. Additionally, a semantic alignment optimization mechanism is introduced to align cross-modal features within a shared semantic space, improving their consistency and complementarity. Extensive experiments on three benchmark datasets CREMA-D, UrbanSound8K-AV, and MNISTDVS-NTIDIGITS demonstrate that S-CMRL significantly outperforms existing multimodal SNN methods, achieving the state-of-the-art performance. The code is publicly available at https://github.com/Brain-Cog-Lab/S-CMRL.

Figures (9)

• Figure 1: Different cross-modal fusion methods in spiking neural networks. (a) Direct fusion, which typically sums the features from different modalities directly. (b) Fusion with cross-modal attention mechanisms. (c) Our proposed semantic-alignment cross-modal residual learning fusion. "Q": Query embedding; "K": Key embedding; "V": Value embedding.
• Figure 2: Experimental results of the Spiking Transformer on the CRMEA-D dataset. CMRL represents Cross-Modal Residual Learning. Due to the weaker visual signals compared to audio in CREMA-D, traditional cross-modal fusion strategies show limited improvement. When incorporating cross-modal features as residuals into the unimodal representations, model performance improves. This highlights the importance of preserving the unimodal-specific semantic features.
• Figure 3: Schematic of cross-modal complementary spatio-temporal spiking attention, using the computation process of the complementary feature $\boldsymbol{x}^a_{\text{res}}$ in audio features as an example. Best viewed in color.
• Figure 4: Overview of proposed semantic-alignment cross-modal residual learning framework. The network processes visual and auditory inputs through independent pathways. Following positional embedding, These pathways converge in a central module, which employs a novel cross-modal complementary spatiotemporal spike attention mechanism. This mechanism effectively exploits complementary information between modalities and integrates it as residuals into the unique feature representations of each modality. Additionally, the semantic alignment optimization further enhances the consistency of cross-modal features.
• Figure 5: Visualization of visual and audio data under different noise intensity in CRMEA-D dataset. The vertical coordinate of each sub-figure represents the SNR value, the smaller the value, the higher the noise intensity. From top to bottom, the graphs present the variation of noise intensity from low to high (30 to 0). The horizontal coordinates show three different emotion categories: Neutral, Happy and Disgust.