Time Cell Inspired Temporal Codebook in Spiking Neural Networks for Enhanced Image Generation

TL;DR

The paper tackles the challenge of temporally coherent generative modeling with spiking neural networks by introducing a hippocampal time cell–inspired temporal codebook within a Spiking VQ-VAE. It combines spike-based encoding, a discrete temporal latent space, and autoregressive generation via transformers (with both spiking and non-spiking variants) to produce high-quality, temporally consistent images across diverse datasets, including high-resolution neuromorphic data. Empirical results show state-of-the-art performance among SNN-based generative models and clear benefits from incorporating temporal information, supported by ablation and destruction studies that highlight the codebook’s critical role. The work underscores the importance of temporal dynamics in neuromorphic generation and points to scalable, temporally aware SNN architectures as a promising direction for future research.

Abstract

This paper presents a novel approach leveraging Spiking Neural Networks (SNNs) to construct a Variational Quantized Autoencoder (VQ-VAE) with a temporal codebook inspired by hippocampal time cells. This design captures and utilizes temporal dependencies, significantly enhancing the generative capabilities of SNNs. Neuroscientific research has identified hippocampal "time cells" that fire sequentially during temporally structured experiences. Our temporal codebook emulates this behavior by triggering the activation of time cell populations based on similarity measures as input stimuli pass through it. We conducted extensive experiments on standard benchmark datasets, including MNIST, FashionMNIST, CIFAR10, CelebA, and downsampled LSUN Bedroom, to validate our model's performance. Furthermore, we evaluated the effectiveness of the temporal codebook on neuromorphic datasets NMNIST and DVS-CIFAR10, and demonstrated the model's capability with high-resolution datasets such as CelebA-HQ, LSUN Bedroom, and LSUN Church. The experimental results indicate that our method consistently outperforms existing SNN-based generative models across multiple datasets, achieving state-of-the-art performance. Notably, our approach excels in generating high-resolution and temporally consistent data, underscoring the crucial role of temporal information in SNN-based generative modeling.

Figures (7)

• Figure 1: The overview of the proposed method. After spike encoding, the input stimulus $\tilde{I}_t$, is processed by the encoder to yield $z_t$. Concatenating $z_t$ and passing it through the temporal codebook $\tilde{Q}$ produces quantized $z^q_t$, which is then input to the decoder to reconstruct the stimulus $X_t$.
• Figure 2: Spatiotemporal trajectories of neural activations from temporal codebook
• Figure 3: Comparative visualization of generated samples across various datasets
• Figure 4: Temporal unfolding and generation quality comparison on neuromorphic datasets using temporal and vanilla codebooks on N-MNIST (a) and DVS-CIFAR10 (b).
• Figure 5: Generated high-resolution images using the proposed SNN-based method