Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hybrid Spiking-Convolutional Neural Network Approach for Advancing Machine Learning Models

Sanaullah, Kaushik Roy, Ulrich Rückert, Thorsten Jungeblut

TL;DR

The paper addresses image inpainting by integrating temporal dynamics through spiking neurons into a convolutional framework. It introduces a standalone hybrid SC-NN with one SNNConv2d (LIF) layer and five CNNConv2d layers, trained on a masked version of the LSDIR dataset using mean squared error losses and the Adam optimizer. The LIF dynamics are described by $\frac{dV(t)}{dt} = \frac{I(t) - V(t)/R}{C}$ with spike generation at $V_{\rm th}$ and reset, and leakage modeled by $\frac{dV}{dt} = -\frac{V}{\tau}$, enabling temporal context in addition to spatial features. Results show a training MSE of $0.015$ and a validation MSE of $0.0017$, with qualitative inpainting improvements over state-of-the-art baselines, illustrating the value of fusing temporal-spiking processing with conventional CNN features for vision tasks.

Abstract

In this article, we propose a novel standalone hybrid Spiking-Convolutional Neural Network (SC-NN) model and test on using image inpainting tasks. Our approach uses the unique capabilities of SNNs, such as event-based computation and temporal processing, along with the strong representation learning abilities of CNNs, to generate high-quality inpainted images. The model is trained on a custom dataset specifically designed for image inpainting, where missing regions are created using masks. The hybrid model consists of SNNConv2d layers and traditional CNN layers. The SNNConv2d layers implement the leaky integrate-and-fire (LIF) neuron model, capturing spiking behavior, while the CNN layers capture spatial features. In this study, a mean squared error (MSE) loss function demonstrates the training process, where a training loss value of 0.015, indicates accurate performance on the training set and the model achieved a validation loss value as low as 0.0017 on the testing set. Furthermore, extensive experimental results demonstrate state-of-the-art performance, showcasing the potential of integrating temporal dynamics and feature extraction in a single network for image inpainting.

A Hybrid Spiking-Convolutional Neural Network Approach for Advancing Machine Learning Models

TL;DR

The paper addresses image inpainting by integrating temporal dynamics through spiking neurons into a convolutional framework. It introduces a standalone hybrid SC-NN with one SNNConv2d (LIF) layer and five CNNConv2d layers, trained on a masked version of the LSDIR dataset using mean squared error losses and the Adam optimizer. The LIF dynamics are described by with spike generation at and reset, and leakage modeled by , enabling temporal context in addition to spatial features. Results show a training MSE of and a validation MSE of , with qualitative inpainting improvements over state-of-the-art baselines, illustrating the value of fusing temporal-spiking processing with conventional CNN features for vision tasks.

Abstract

In this article, we propose a novel standalone hybrid Spiking-Convolutional Neural Network (SC-NN) model and test on using image inpainting tasks. Our approach uses the unique capabilities of SNNs, such as event-based computation and temporal processing, along with the strong representation learning abilities of CNNs, to generate high-quality inpainted images. The model is trained on a custom dataset specifically designed for image inpainting, where missing regions are created using masks. The hybrid model consists of SNNConv2d layers and traditional CNN layers. The SNNConv2d layers implement the leaky integrate-and-fire (LIF) neuron model, capturing spiking behavior, while the CNN layers capture spatial features. In this study, a mean squared error (MSE) loss function demonstrates the training process, where a training loss value of 0.015, indicates accurate performance on the training set and the model achieved a validation loss value as low as 0.0017 on the testing set. Furthermore, extensive experimental results demonstrate state-of-the-art performance, showcasing the potential of integrating temporal dynamics and feature extraction in a single network for image inpainting.
Paper Structure (8 sections, 3 equations, 3 figures, 2 tables)

This paper contains 8 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Discussion and Results
  3. Temporal Dynamics
  4. Proposed Hybrid Approach
  5. LIF Neural Model
  6. Dataset
  7. Experimental Results
  8. Conclusion

Figures (3)

  • Figure 1: The image shows the architecture of the proposed hybrid model for image inpainting. It consists of six layers, five of which are regular CNNConv2d layers responsible for standard convolutional operations. The one layer, SNNConv2d, utilizes the LIF neural mode to introduce spiking behavior into the network. ReLU activation functions are applied after each layer to enhance non-linearity and feature extraction. The model effectively combines the advantages of spiking neural processing and conventional convolutional operations, making it well-suited for image inpainting tasks by capturing both spatial and temporal information.
  • Figure 2: The basic graphical structural representation of LIF neural model. (a) temporal dynamics of membrane potential ($V_t$) and the input current ($I_t$) at time $t$ in the post-neuron. (b) equivalent circuit model of the LIF neuron.
  • Figure 3: The test images containing a missing or corrupted region and the model attempt to restore the missing part of the image using the hybrid SC-NN approach.