Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpikeVAEDiff: Neural Spike-based Natural Visual Scene Reconstruction via VD-VAE and Versatile Diffusion

Jialu Li, Taiyan Zhou

TL;DR

SpikeVAEDiff addresses high-resolution reconstruction of natural visual scenes from neural spike data by integrating a Very Deep Variational Autoencoder (VDVAE) with a diffusion-based refinement driven by multimodal CLIP features. The method operates in two stages: first mapping spikes to a VDVAE latent space to obtain a coarse $64×64$ image, then conditioning Versatile Diffusion on predicted CLIP-Vision and CLIP-Text features (with BLIP2 captions) to produce a high-resolution $512×512$ image, all without finetuning pretrained components. Evaluated on the Allen Visual Coding—Neuropixels dataset, the approach reveals that VISI activity most strongly supports reconstruction quality and demonstrates that spike data offer superior temporal/spatial resolution compared to fMRI baselines, with ablations confirming the importance of including multiple visual regions. Overall, SpikeVAEDiff showcases a practical pathway to combine pretrained generative priors with neural signals for brain-computer interface applications, while highlighting region-specific roles and future potential for multi-modal decoding and BMI use cases.

Abstract

Reconstructing natural visual scenes from neural activity is a key challenge in neuroscience and computer vision. We present SpikeVAEDiff, a novel two-stage framework that combines a Very Deep Variational Autoencoder (VDVAE) and the Versatile Diffusion model to generate high-resolution and semantically meaningful image reconstructions from neural spike data. In the first stage, VDVAE produces low-resolution preliminary reconstructions by mapping neural spike signals to latent representations. In the second stage, regression models map neural spike signals to CLIP-Vision and CLIP-Text features, enabling Versatile Diffusion to refine the images via image-to-image generation. We evaluate our approach on the Allen Visual Coding-Neuropixels dataset and analyze different brain regions. Our results show that the VISI region exhibits the most prominent activation and plays a key role in reconstruction quality. We present both successful and unsuccessful reconstruction examples, reflecting the challenges of decoding neural activity. Compared with fMRI-based approaches, spike data provides superior temporal and spatial resolution. We further validate the effectiveness of the VDVAE model and conduct ablation studies demonstrating that data from specific brain regions significantly enhances reconstruction performance.

SpikeVAEDiff: Neural Spike-based Natural Visual Scene Reconstruction via VD-VAE and Versatile Diffusion

TL;DR

SpikeVAEDiff addresses high-resolution reconstruction of natural visual scenes from neural spike data by integrating a Very Deep Variational Autoencoder (VDVAE) with a diffusion-based refinement driven by multimodal CLIP features. The method operates in two stages: first mapping spikes to a VDVAE latent space to obtain a coarse image, then conditioning Versatile Diffusion on predicted CLIP-Vision and CLIP-Text features (with BLIP2 captions) to produce a high-resolution image, all without finetuning pretrained components. Evaluated on the Allen Visual Coding—Neuropixels dataset, the approach reveals that VISI activity most strongly supports reconstruction quality and demonstrates that spike data offer superior temporal/spatial resolution compared to fMRI baselines, with ablations confirming the importance of including multiple visual regions. Overall, SpikeVAEDiff showcases a practical pathway to combine pretrained generative priors with neural signals for brain-computer interface applications, while highlighting region-specific roles and future potential for multi-modal decoding and BMI use cases.

Abstract

Reconstructing natural visual scenes from neural activity is a key challenge in neuroscience and computer vision. We present SpikeVAEDiff, a novel two-stage framework that combines a Very Deep Variational Autoencoder (VDVAE) and the Versatile Diffusion model to generate high-resolution and semantically meaningful image reconstructions from neural spike data. In the first stage, VDVAE produces low-resolution preliminary reconstructions by mapping neural spike signals to latent representations. In the second stage, regression models map neural spike signals to CLIP-Vision and CLIP-Text features, enabling Versatile Diffusion to refine the images via image-to-image generation. We evaluate our approach on the Allen Visual Coding-Neuropixels dataset and analyze different brain regions. Our results show that the VISI region exhibits the most prominent activation and plays a key role in reconstruction quality. We present both successful and unsuccessful reconstruction examples, reflecting the challenges of decoding neural activity. Compared with fMRI-based approaches, spike data provides superior temporal and spatial resolution. We further validate the effectiveness of the VDVAE model and conduct ablation studies demonstrating that data from specific brain regions significantly enhances reconstruction performance.
Paper Structure (18 sections, 4 equations, 6 figures)

This paper contains 18 sections, 4 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Neural Signals for Image Reconstruction
  3. Generative Models in Image Reconstruction
  4. This Work
  5. Contributions
  6. Related Work
  7. Methods
  8. Overview
  9. First Stage: Low-Level Reconstruction of Images using VDVAE
  10. Second Stage: Final Reconstruction of Images using Versatile Diffusion
  11. Experiments
  12. Dataset
  13. Activation of Different Brain Regions in Response to Stimuli
  14. Image Reconstruction Overview
  15. Comparison with State-of-the-Art in fMRI
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Dataset overview.
  • Figure 2: Peristimulus Time Histograms for different brain regions on stimulus.
  • Figure 3: Examples of spikes reconstructions from our model.
  • Figure 4: Failure cases of spikes reconstructions from our model.
  • Figure 5: Comparison of reconstructions by VAE and VdVAE.
  • ...and 1 more figures