Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

Jiangrong Shen, Yulin Xie, Qi Xu, Gang Pan, Huajin Tang, Badong Chen

TL;DR

This work tackles modality imbalance and temporal misalignment in multimodal spiking neural networks by introducing a Temporal Attention-guided Adaptive Fusion (TAAF) module and a temporal adaptive balanced fusion loss. The per-timestep attention scores guide both feature fusion and loss weighting, while gradient modulation balances learning rates across modalities to prevent dominance. Empirical results on CREMA-D, AVE, and EAD show state-of-the-art accuracy with substantially fewer time steps and improved energy efficiency, validating the approach's ability to leverage temporal dynamics for robust multisensory integration. The method advances neuromorphic multimodal learning by aligning temporal representations with biological principles and enabling practical deployment on energy-constrained hardware.

Abstract

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55\%, 70.65\% and 97.5\%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.

Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

TL;DR

This work tackles modality imbalance and temporal misalignment in multimodal spiking neural networks by introducing a Temporal Attention-guided Adaptive Fusion (TAAF) module and a temporal adaptive balanced fusion loss. The per-timestep attention scores guide both feature fusion and loss weighting, while gradient modulation balances learning rates across modalities to prevent dominance. Empirical results on CREMA-D, AVE, and EAD show state-of-the-art accuracy with substantially fewer time steps and improved energy efficiency, validating the approach's ability to leverage temporal dynamics for robust multisensory integration. The method advances neuromorphic multimodal learning by aligning temporal representations with biological principles and enabling practical deployment on energy-constrained hardware.

Abstract

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55\%, 70.65\% and 97.5\%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.

Paper Structure

This paper contains 13 sections, 10 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal learning in SNNs
  4. Multimodal learning methods for imbalanced problem
  5. Method
  6. Temporal Attention-guided Adaptive Fusion Module
  7. Temporal Adaptive Balanced Fusion Loss
  8. Results
  9. Performance Comparison
  10. Ablation Study
  11. Modality Imbalance Analysis
  12. Conclusion
  13. Acknowledgements

Figures (2)

  • Figure 1: Architecture of the proposed attention-guided multimodal SNN. The temporal attention-guided adaptive fusion (TAAF) module generates temporal importance scores at every timestep that modulate both feature fusion and loss calculation. The temporal importance scores are then integrated into the temporal adaptive balanced fusion loss to enable dynamic adjustment for each modality branch in individual timesteps.
  • Figure 2: The epoch-wise analysis of accuracy and loss trajectories with the proposed multimodal SNNs on CREMA-D and AVE datasets. (a) The top and bottom figures are the accuracy and loss trajectories for CREMA-D datasets, respectively. (b) The top and bottom figures are the accuracy and loss trajectories for AVE datasets, respectively. (c) The accuracy trajectory of the baseline model without the proposed TAAF. (4) The accuracy trajectory of our model without time step alignment module.