SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network

TL;DR

SpikeCLIP tackles multimodal integration in spiking neural networks by pairing a dual-stream SNN with a two-stage training regimen: cross-modal alignment through knowledge distillation from CLIP, followed by dual-loss fine-tuning with a KL regularizer to preserve generalization. The approach achieves competitive image classification and zero-shot performance while delivering substantial energy savings on neuromorphic hardware-like computations. Through extensive ablations and architecture choices (e.g., Spikingformer image backbone and MLP text encoder), the work demonstrates the viability of energy-efficient, biologically plausible multimodal learning with SNNs and highlights avenues for scaling to larger datasets and generative tasks.

Abstract

Spiking Neural Networks (SNNs) have emerged as a promising alternative to conventional Artificial Neural Networks (ANNs), demonstrating comparable performance in both visual and linguistic tasks while offering the advantage of improved energy efficiency. Despite these advancements, the integration of linguistic and visual features into a unified representation through spike trains poses a significant challenge, and the application of SNNs to multimodal scenarios remains largely unexplored. This paper presents SpikeCLIP, a novel framework designed to bridge the modality gap in spike-based computation. Our approach employs a two-step recipe: an ``alignment pre-training'' to align features across modalities, followed by a ``dual-loss fine-tuning'' to refine the model's performance. Extensive experiments reveal that SNNs achieve results on par with ANNs while substantially reducing energy consumption across various datasets commonly used for multimodal model evaluation. Furthermore, SpikeCLIP maintains robust image classification capabilities, even when dealing with classes that fall outside predefined categories. This study marks a significant advancement in the development of energy-efficient and biologically plausible multimodal learning systems. Our code is available at https://github.com/Lvchangze/SpikeCLIP.

Figures (6)

• Figure 1: The training framework of SpikeCLIP.
• Figure 2: The architecture of SpikeCLIP.
• Figure 3: An illustration of our two-step training approach for multimodal SNNs. First, we pre-train SpikeCLIP by distilling knowledge from conventional CLIP, using a readout layer to map SNN states to floating-point feature representations. Then, we fine-tune SpikeCLIP on downstream datasets, adding a regularization term based on Kullback-Leibler divergence on the training loss.
• Figure 4: Overview of the architecture of SpikeCLIP. We use dual-stream architectures for multimodal modeling. The spiking image encoder is based on Spikingformer zhou2023spikingformer, while a simple spiking MLP is designed for the text encoder of SpikeCLIP, with integrate-and-fire neurons converting data into spike trains for SNN processing.
• Figure 5: Zero-shot results on $6$ image classification datasets. SpikeCLIP achieved results comparable to ScratchCLIP, with a negligible average difference of only $0.72\%$.