Late Breaking Results: Quamba-SE: Soft-edge Quantizer for Activations in State Space Models

TL;DR

Quamba-SE tackles the outlier-heavy activation quantization challenge in State Space Models by introducing a hardware-friendly soft-edge quantizer with three adaptive scales, enabling high precision for small values, standard precision for normal values, and low precision for outliers. The method leverages thresholds $L$ and $H$ and uses an additional INT8 bit to distinguish value regions, improving dynamic range without extra storage. Evaluations on the 130M Mamba model across six zero-shot benchmarks show consistent improvements over Quamba, with dataset gains up to +2.68% and average gains up to +0.83%, demonstrating the practicality of hardware-level soft-edge quantization for SSMs. The work suggests promising directions for hardware synthesis and layer-specific quantization strategies in compact LLM deployments on edge devices.

Abstract

We propose Quamba-SE, a soft-edge quantizer for State Space Model (SSM) activation quantization. Unlike existing methods, using standard INT8 operation, Quamba-SE employs three adaptive scales: high-precision for small values, standard scale for normal values, and low-precision for outliers. This preserves outlier information instead of hard clipping, while maintaining precision for other values. We evaluate on Mamba- 130M across 6 zero-shot benchmarks. Results show that Quamba- SE consistently outperforms Quamba, achieving up to +2.68% on individual benchmarks and up to +0.83% improvement in the average accuracy of 6 datasets.

Figures (2)

• Figure 1: Quamba-SE: dataflow and data precision
• Figure 2: Hardware architecture of Quamba-SE