EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices

TL;DR

Deploying large language models on edge devices is constrained by memory bandwidth and storage. EntroLLM combines a mixed tensor-level quantization (layer-specific unsigned or asymmetric) with entropy coding using Huffman encoding and a parallel decoding strategy, preserving accuracy without retraining. It delivers substantial gains, including up to 8.1x and 13.1x improvements in downstream entropy compressibility for 8-bit and 4-bit weights, respectively, along with storage savings up to 30% over uint8 and 65% over uint4, and latency reductions up to 146.6% on edge hardware. This approach enables practical, low-power edge-LMM deployment while remaining compatible with existing post-training quantization pipelines.

Abstract

Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by $7\times$ (8-bit) and $11.3\times$ (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to $30\%$ storage savings over uint8 and $65\%$ over uint4 models, with $31.9-146.6\%$ faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment.

Figures (4)

• Figure 1: Overall schematic of our edge-device inference scheme. A floating-point model is first quantized using a mixed scheme to maximize compressibility. For storage on edge devices, integer weights are entropy-coded, then decoded in parallel during inference to minimize latency. Section \ref{['sec:method']} details these components.
• Figure 2: A visual explanation of the different uniform quantization grids nagel2021whitepaperneuralnetwork for a bit-width of $8$. $s$ is the scaling factor, $z$ the zero-point. The floating-point grid is in black, and the integer quantized grid is in blue. In our work, we use either an unsigned or an asymmetric quantization scheme on each layer based on the individual layer's weight distribution.
• Figure 3: In the serial decoding setting, parallelization is not possible as the variable code lengths make it difficult for us to predict the start token of a symbol in our remaining encoded parameters. By keeping the parameter space's original weight tensor packing structure intact, we can parallelize decoding, as now we can assign different encoded tensors (chunks) to individual threads assigned to cores. Modern LLMs have hundreds to thousands of such weight tensors, and hence, coarse-grained parallelism is achievable. In the figure <p> represents current bits being decoded, while <unk> represents unknown encoded parameter bits.
• Figure 4: Top: Parameter distribution for 8-bit quantized models. Bottom: Parameter distribution for 4-bit counterparts. Model size increases from left to right.