SVL: Spike-based Vision-language Pretraining for Efficient 3D Open-world Understanding

TL;DR

SVL addresses the energy-efficiency gap between Spiking Neural Networks and Artificial Neural Networks for open-world 3D understanding by pretraining spike-based encoders with a vision-language objective. It introduces Multi-scale Triple Alignment (MTA) to jointly align 3D, image, and text modalities, and Rep-VLI to enable lightweight inference without relying on large text encoders. Leveraging CLIP as a semantic space, SVL achieves state-of-the-art zero-shot 3D classification (e.g., $85.4\%$ on ModelNet40) and consistently improves downstream 3D tasks, while delivering substantial energy savings thanks to spike-based processing and Rep-VLI. The framework also demonstrates open-world 3D generative captioning and question answering, signaling a practical convergence of neuromorphic efficiency and multimodal generalization for robust 3D perception in resource-constrained environments.

Abstract

Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. However, existing SNNs still exhibit a significant performance gap compared to Artificial Neural Networks (ANNs) due to inadequate pre-training strategies. These limitations manifest as restricted generalization ability, task specificity, and a lack of multimodal understanding, particularly in challenging tasks such as multimodal question answering and zero-shot 3D classification. To overcome these challenges, we propose a Spike-based Vision-Language (SVL) pretraining framework that empowers SNNs with open-world 3D understanding while maintaining spike-driven efficiency. SVL introduces two key components: (i) Multi-scale Triple Alignment (MTA) for label-free triplet-based contrastive learning across 3D, image, and text modalities, and (ii) Re-parameterizable Vision-Language Integration (Rep-VLI) to enable lightweight inference without relying on large text encoders. Extensive experiments show that SVL achieves a top-1 accuracy of 85.4% in zero-shot 3D classification, surpassing advanced ANN models, and consistently outperforms prior SNNs on downstream tasks, including 3D classification (+6.1%), DVS action recognition (+2.1%), 3D detection (+1.1%), and 3D segmentation (+2.1%) with remarkable efficiency. Moreover, SVL enables SNNs to perform open-world 3D question answering, sometimes outperforming ANNs. To the best of our knowledge, SVL represents the first scalable, generalizable, and hardware-friendly paradigm for 3D open-world understanding, effectively bridging the gap between SNNs and ANNs in complex open-world understanding tasks. Code is available https://github.com/bollossom/SVL.

Figures (3)

• Figure 1: Overall architecture and applications of our SVL. (a) In pretraining, we proposed Multi-scale Triple Alignment (MTA) that jointly optimizes correlation alignment across text, image, and 3D inputs. (b) For downstream tasks, we propose Re-parameterizable Vision-Language Integration (Rep-VLI) to reparameterize the text embeddings generated by the text encoder into lightweight weights, enabling efficient spike-driven inference.
• Figure 2: Dialogues between SVL-13B and a human user. The dialogues show SVL’s ability to understand point clouds’ shapes, appearances, functionalities, etc. Additionally, SVL demonstrates abilities to respond to human instructions with common sense, avoiding biases.
• Figure 3: Architecture details of open-world multimodel learning.