ELUTQ: Efficient LUT-Aware Quantization for Deploying Large Language Models on Edge Devices

TL;DR

ELUTQ introduces Hierarchical Linear Quantization (HLQ), a non-uniform weight quantization designed to better fit weight distributions and eliminate dequantization overhead when paired with Bit-serial LUT-based GEMM for edge CPUs. HLQ is orthogonal to existing PTQ and efficient finetuning methods, and is complemented by a pure-C++ edge inference runtime, weight-rearrangement, mirror storage, and table-based quantization to minimize memory and latency. The framework supports post-training quantization (HLQ-GPTQ) and efficient finetuning (HLQ-Finetuning), achieving substantial perplexity improvements at 2–3 bit and enabling quantization of extremely large models (e.g., LLaMA3.1-70B) with modest CPU/GPU resources via disk offload. On edge devices such as Apple M2 and RK3588, ELUTQ delivers competitive or superior end-to-end throughput compared to baselines, thanks to optimized LUT-based GEMM and careful system-level optimizations, demonstrating practical deployment of large language models on commodity hardware.

Abstract

Weight quantization effectively reduces memory consumption and enables the deployment of large language models on CPU-based edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit settings. In this paper, we propose ELUTQ, an efficient quantization framework featuring a novel quantization format termed Hierarchical Linear Quantization (HLQ). HLQ is designed to better capture the statistical characteristics of weights without increasing the computational cost of bit-serial LUT-based GEMM operations, thereby eliminating dequantization overhead. HLQ is orthogonal to existing quantization algorithms. For the LLaMA3.1-8B model, when combined with post-training quantization, HLQ improves uniform quantization by achieving approximately 8 percent perplexity reduction at 3-bit precision and 85 percent perplexity reduction at 2-bit precision. When combined with efficient finetuning techniques, HLQ further improves model accuracy. We also integrate a disk-offload technique into ELUTQ, enabling it to complete the quantization of LLaMA3.1-70B using only 64 GB of CPU memory and 48 GB of VRAM, significantly reducing the hardware requirements for large-scale model quantization. To enable efficient deployment on edge devices, ELUTQ provides high-performance CPU kernels to support end-to-end inference. Under a 4-thread configuration with batch size 1, our 2-bit quantized LLaMA2-7B model achieves a throughput of more than 25 tokens per second on an Apple M2 chip. All the code is available at https://github.com/Nkniexin/ELUTQ.

Figures (11)

• Figure 1: Overview of ELUTQ.
• Figure 2: Illustration of three computation paradigms for weight-only quantized matrix multiplication.
• Figure 3: Detail pipeline of Bit-serial Lut-based GEMM.
• Figure 4: (Top) The weight distribution of one output channel in a up_proj of LLaMA3.1-8B. (Bottom) MSE loss for weight comparsion between Uniform and HLQ.
• Figure 5: Standard GPTQ VS HLQ-GPTQ.