MTSA-SNN: A Multi-modal Time Series Analysis Model Based on Spiking Neural Network

TL;DR

MTSA-SNN introduces a multi-modal, event-driven framework for time-series analysis based on Spiking Neural Networks. It unifies temporal images and sequential data into pulse signals via a Single-Modal Pulse Encoding Module, fuses heterogeneous pulses with a Joint Learning Module that leverages a learnable function $\Psi$ and a probability distribution $P_{mtsa}$, and enhances temporal analysis through Wavelet Transform decomposition into subbands $LL$, $LH$, $HH$, and $HL$. The model is validated on MIT-BIH Arrhythmia, ETT, and stock market datasets, achieving state-of-the-art classification and regression performance, with ablation studies confirming the benefits of joint modality fusion and wavelet-based processing. By delivering a scalable, event-driven approach to complex temporal information, MTSA-SNN offers a practical framework for robust time-series analysis with potential impact on domains requiring real-time, multi-modal interpretation of dynamic signals.

Abstract

Time series analysis and modelling constitute a crucial research area. Traditional artificial neural networks struggle with complex, non-stationary time series data due to high computational complexity, limited ability to capture temporal information, and difficulty in handling event-driven data. To address these challenges, we propose a Multi-modal Time Series Analysis Model Based on Spiking Neural Network (MTSA-SNN). The Pulse Encoder unifies the encoding of temporal images and sequential information in a common pulse-based representation. The Joint Learning Module employs a joint learning function and weight allocation mechanism to fuse information from multi-modal pulse signals complementary. Additionally, we incorporate wavelet transform operations to enhance the model's ability to analyze and evaluate temporal information. Experimental results demonstrate that our method achieved superior performance on three complex time-series tasks. This work provides an effective event-driven approach to overcome the challenges associated with analyzing intricate temporal information. Access to the source code is available at https://github.com/Chenngzz/MTSA-SNN}{https://github.com/Chenngzz/MTSA-SNN

Figures (6)

• Figure 1: The Overall Structure of MTSA-SNN. SNN Encoder workflow is showed in Algorithm 2.
• Figure 2: Data (wavelet transform) converted into pulse signals by MTSA-SNN (above) & Original data converted into pulse signals by MTSA-SNN
• Figure 3: ETT dataset signal features across different frequency and spatial scales. (LL captures low-frequency signal components. LH and HH capture high-frequency components in both low and high-frequency signals. HL contains low-frequency components of high-frequency signals.)
• Figure 4: Stock prediction dataset signal features across different frequency and spatial scales
• Figure 5: The heatmap of MTSA-SNN's various component neuron activations. Specifically, (A) and (B) represent the neuron activation patterns after the time series information passes through the image encoder and sequence encoder of MTSA-SNN. (C) demonstrates the fused output after the joint learning process for the original temporal information. (D) represents the pulse fusion after applying wavelet transform in MTSA-SNN.