LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models

TL;DR

This work addresses the challenge of converting large language models into fully spike-driven networks without accuracy loss. It introduces two novel neurons, the Outlier-Aware Threshold (OAT) and Hierarchically Gated (HG) neuron, to handle activation outliers and nonlinearities, respectively, and provides spike-equivalent implementations for self-attention, FFN, LayerNorm, and Softmax. The approach demonstrates near-lossless conversion across diverse models (up to OPT-66B) and tasks, with notable gains in energy efficiency and even some performance improvements on select benchmarks. The results suggest that high-performance, fully spike-driven LLMs are feasible at commercially relevant scales, with practical benefits for neuromorphic hardware and energy-constrained deployments. The work also highlights the importance of pre-trained model quality for multimodal tasks when transferring to SNNs.

Abstract

Spiking Large Language Models (LLMs) have emerged as an energy-efficient alternative to conventional LLMs through their event-driven computation. To effectively obtain spiking LLMs, researchers develop different ANN-to-SNN conversion methods by leveraging pre-trained ANN parameters while inheriting the energy efficiency of SNN. However, existing conversion methods struggle with extreme activation outliers and incompatible nonlinear operations of ANN-based LLMs. To address this, we propose a loss-less ANN-SNN conversion for fully spike-driven LLMs, termed LAS. Specifically, LAS introduces two novel neurons to convert the activation outlier and nonlinear operation of ANN-based LLMs. Moreover, LAS tailors the spike-equivalent Transformer components for spiking LLMs, which can ensure full spiking conversion without any loss of performance. Experimental results on six language models and two vision-language models demonstrate that LAS achieves loss-less conversion. Notably, on OPT-66B, LAS even improves the accuracy of 2\% on the WSC task. In addition, the parameter and ablation studies further verify the effectiveness of LAS. The source code is available at https://github.com/lc783/LAS

Figures (6)

• Figure 1: Visualizations of outliers on OPT-7B. (a) Extensive outliers from attention mechanism. (b) The information loss of the converted activations.
• Figure 2: The overview of the proposed LAS method. OAT and HG neurons are designed to convert activation outliers and nonlinear operations of ANN-based LLMs, respectively. $n$ and $d$ denote the number of tokens and the channel dimensions, respectively.
• Figure 3: An approximated for GELU with time step=16.
• Figure 4: An approximated for exponent with time step=16.
• Figure 5: Ablations on components in STSB task