QSLM: A Performance- and Memory-aware Quantization Framework with Tiered Search Strategy for Spike-driven Language Models

TL;DR

The paper tackles the problem of quantizing pre-trained Spike-driven Language Models (SLMs) to fit tight performance and memory budgets for embedded deployment. It introduces QSLM, a framework that automates quantization using network analysis, a tiered search strategy, and a quantization-setting selection mechanism guided by a multi-objective trade-off that balances accuracy or perplexity with memory usage, incorporating a parameter $\alpha$. The approach emphasizes architecture awareness by identifying quantization-sensitive blocks and applying global-, block-, and module-level quantization, with model selection driven by a score that respects $A_{acc}$, $A_{ppx}$, and memory ratio $M_q/M$. Empirical results show memory footprint reductions up to 86.5% and power savings up to 20%, while maintaining SST-2 accuracy and WikiText-2 perplexity close to the baseline, demonstrating meaningful potential for embedded SLM deployment and design automation. $A_{acc}$ and $A_{ppx}$ are used to quantify task performance, while $N_T$ denotes token count for perplexity calculations, and the trade-off leverages $\alpha$ to prioritize memory vs. performance as needed.

Abstract

Large Language Models (LLMs) have been emerging as prominent AI models for solving many natural language tasks due to their high performance (e.g., accuracy) and capabilities in generating high-quality responses to the given inputs. However, their large computational cost, huge memory footprints, and high processing power/energy make it challenging for their embedded deployments. Amid several tinyLLMs, recent works have proposed spike-driven language models (SLMs) for significantly reducing the processing power/energy of LLMs. However, their memory footprints still remain too large for low-cost and resource-constrained embedded devices. Manual quantization approach may effectively compress SLM memory footprints, but it requires a huge design time and compute power to find the quantization setting for each network, hence making this approach not-scalable for handling different networks, performance requirements, and memory budgets. To bridge this gap, we propose QSLM, a novel framework that performs automated quantization for compressing pre-trained SLMs, while meeting the performance and memory constraints. To achieve this, QSLM first identifies the hierarchy of the given network architecture and the sensitivity of network layers under quantization, then employs a tiered quantization strategy (e.g., global-, block-, and module-level quantization) while leveraging a multi-objective performance-and-memory trade-off function to select the final quantization setting. Experimental results indicate that our QSLM reduces memory footprint by up to 86.5%, reduces power consumption by up to 20%, maintains high performance across different tasks (i.e., by up to 84.4% accuracy of sentiment classification on the SST-2 dataset and perplexity score of 23.2 for text generation on the WikiText-2 dataset) close to the original non-quantized model while meeting the performance and memory constraints.

Figures (10)

• Figure 1: Current trends of performance (i.e., accuracy), number of weight parameters (note, M denotes millions [10$^6$] of parameters), and energy consumption of SLMs Ref_Xing_SpikeLM_ICML24Ref_Bal_SpikingBERT_AAAI24Ref_Su_SNNBERT_NeuNet24Ref_Xing_SpikeLLM_ICLR24Ref_Chu_SpikeGPT_TMLR24 on the sentiment analysis task with the SST-2 dataset Ref_Wang_GLUE_ICLR2019.
• Figure 2: Performance profiles of the pre-trained SpikeGPT-216M after uniformly quantizing its weight parameters in its attention blocks across different precision levels for different tasks: (a) sentiment classification on the SST-2 dataset Ref_Wang_GLUE_ICLR2019, and (b) perplexity on the WikiText-2 dataset Ref_Merity_WikiText_ICLR17. Note, a lower perplexity score represents a better text generation performance.
• Figure 3: Overview of our novel contributions.
• Figure 4: Overview of the SpikeGPT architecture. $B$ is the number of attention blocks. For instance, the pre-trained SpikeGPT-216M has $B$=18 blocks Ref_Chu_SpikeGPT_TMLR24.
• Figure 5: Our QSLM framework showing its key steps: network model analysis, tiered search strategy for quantization, and quantization setting selection.