Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Extreme Quantization in Spiking Language Models

Malyaban Bal, Yi Jiang, Abhronil Sengupta

TL;DR

This paper addresses the energy and power bottlenecks of large language models by introducing a spiking language model engineered with extreme quantization. It develops a 1/1.58-bit spiking LM based on a BERT-like encoder, trained via equilibrium-based learning and knowledge distillation from a full-precision teacher to preserve accuracy. The key contributions include architecture and learning dynamics for quantized spiking LMs, explicit 1-bit and 1.58-bit weight quantization schemes, a KD framework that leverages steady-state ASR of intermediate layers, and GLUE-based results showing near full-precision performance with substantial energy savings. The approach enables deployment on neuromorphic or in-memory accelerators, offering a path toward edge-efficient sequence modeling with tunable accuracy-energy tradeoffs.

Abstract

Despite the growing prevalence of large language model (LLM) architectures, a crucial concern persists regarding their energy and power consumption, which still lags far behind the remarkable energy efficiency of the human brain. Recent strides in spiking language models (LM) and transformer architectures aim to address this concern by harnessing the spiking activity of biological neurons to enhance energy/power efficiency. Doubling down on the principles of model quantization and energy efficiency, this paper proposes the development of a novel binary/ternary (1/1.58-bit) spiking LM architecture. Achieving scalability comparable to a deep spiking LM architecture is facilitated by an efficient knowledge distillation technique, wherein knowledge from a non-spiking full-precision "teacher" model is transferred to an extremely weight quantized spiking "student" LM. Our proposed model represents a significant advancement as the first-of-its-kind 1/1.58-bit spiking LM, and its performance is rigorously evaluated on multiple text classification tasks of the GLUE benchmark.

Exploring Extreme Quantization in Spiking Language Models

TL;DR

This paper addresses the energy and power bottlenecks of large language models by introducing a spiking language model engineered with extreme quantization. It develops a 1/1.58-bit spiking LM based on a BERT-like encoder, trained via equilibrium-based learning and knowledge distillation from a full-precision teacher to preserve accuracy. The key contributions include architecture and learning dynamics for quantized spiking LMs, explicit 1-bit and 1.58-bit weight quantization schemes, a KD framework that leverages steady-state ASR of intermediate layers, and GLUE-based results showing near full-precision performance with substantial energy savings. The approach enables deployment on neuromorphic or in-memory accelerators, offering a path toward edge-efficient sequence modeling with tunable accuracy-energy tradeoffs.

Abstract

Despite the growing prevalence of large language model (LLM) architectures, a crucial concern persists regarding their energy and power consumption, which still lags far behind the remarkable energy efficiency of the human brain. Recent strides in spiking language models (LM) and transformer architectures aim to address this concern by harnessing the spiking activity of biological neurons to enhance energy/power efficiency. Doubling down on the principles of model quantization and energy efficiency, this paper proposes the development of a novel binary/ternary (1/1.58-bit) spiking LM architecture. Achieving scalability comparable to a deep spiking LM architecture is facilitated by an efficient knowledge distillation technique, wherein knowledge from a non-spiking full-precision "teacher" model is transferred to an extremely weight quantized spiking "student" LM. Our proposed model represents a significant advancement as the first-of-its-kind 1/1.58-bit spiking LM, and its performance is rigorously evaluated on multiple text classification tasks of the GLUE benchmark.
Paper Structure (10 sections, 7 equations, 2 figures, 1 table)

This paper contains 10 sections, 7 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Architecture and Learning Dynamics
  4. Quantizing linear layers to 1/1.58-bit
  5. Knowledge Distillation (KD)
  6. Results
  7. Datasets
  8. Experimental Setup
  9. Energy & Power Efficiency
  10. Conclusion & Future Works

Figures (2)

  • Figure 1: High-level architecture of each encoder layer of the 1-bit Spiking transformer architecture with Quantized Linear layers. All linear layers, including those used in the attention module, are quantized to 1-bit.
  • Figure 2: Results obtained after passing a set of randomly sampled inputs from MRPC dataset. (a) The convergence dynamics of the different sub-layers of an encoder layer of the 1-bit SpikingBERT4 model. (b) Comparison of output layer ASR convergence dynamics of full precision (weights) and 1-bit SpikingBERT4. Y-axis in both shows mean (over number of neurons) of the ASR while X-axis shows the operating time steps.