Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification

TL;DR

Sherry targets hardware-efficient edge deployment of LLMs by pushing ternary quantization to a practical 1.25-bit width. It achieves this with a 3:4 structured sparsity that packs four weights into five bits, enabling SIMD-friendly, LUT-based inference, paired with Arenas to prevent training-time weight trapping. The Arenas module introduces a decaying residual path, preserving gradient diversity during QAT and enabling zero-overhead inference after training. Empirical results on LLaMA-3.2 1B/3B show Sherry reaches state-of-the-art parity at a 25% bit reduction and delivers CPU-side speedups around 10–18%, demonstrating a practical path to edge-capable LLMs. This work offers a concrete hardware-aware approach that narrows the gap between ultra-low-bit quantization theory and real-world edge deployment.

Abstract

The deployment of Large Language Models (LLMs) on resource-constrained edge devices is increasingly hindered by prohibitive memory and computational requirements. While ternary quantization offers a compelling solution by reducing weights to {-1, 0, +1}, current implementations suffer from a fundamental misalignment with commodity hardware. Most existing methods must choose between 2-bit aligned packing, which incurs significant bit wastage, or 1.67-bit irregular packing, which degrades inference speed. To resolve this tension, we propose Sherry, a hardware-efficient ternary quantization framework. Sherry introduces a 3:4 fine-grained sparsity that achieves a regularized 1.25-bit width by packing blocks of four weights into five bits, restoring power-of-two alignment. Furthermore, we identify weight trapping issue in sparse ternary training, which leads to representational collapse. To address this, Sherry introduces Arenas, an annealing residual synapse mechanism that maintains representational diversity during training. Empirical evaluations on LLaMA-3.2 across five benchmarks demonstrate that Sherry matches state-of-the-art ternary performance while significantly reducing model size. Notably, on an Intel i7-14700HX CPU, our 1B model achieves zero accuracy loss compared to SOTA baselines while providing 25% bit savings and 10% speed up. The code is available at https://github.com/Tencent/AngelSlim .

Figures (11)

• Figure 1: Comparison of different packing strategies for ternary quantization in efficiency.
• Figure 2: (Left)2-bit strategy packs each weight into 2 bits to maintain alignment, resulting in large bit wastage. (Middle)1.67-bit strategy packs 3 weights into 5 bits, introducing SIMD-unfriendly 3-way patterns, leading to slow speed. (Right)Our Sherry enforces a 3:4 sparsity and packs 4 weights into 5 bits, introducing SIMD-friendly 4-way patterns, achieving a small 1.25-bit width and faster inference speed. More details are shown in Fig \ref{['fig: sys']}.
• Figure 3: Comparison of weight distributions for LLaMA-1B under different quantization schemes. (Left) Naive 3:4 sparse ternary training exhibits weight trapping, where values collapse into a binary-like polarization, leading to a significant accuracy gap compared to dense models. (Right) Our Sherry utilizes the Arenas module to achieve a trap-free distribution, bridging the performance gap to dense ternary models.
• Figure 4: The effective ranks (ER) of the gradients during training. Both Binary and 3:4 sparse ternary training have relatively low ER due to gradient homogenization.
• Figure 5: The overview of the Arenas module with QAT. The Arenas module injects the heterogeneous gradients through a residual synapse with an annealing gate.