Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

TL;DR

This work tackles the practical challenge of deploying multiple LLMs with different sizes under memory and cost constraints on edge devices. It introduces any-precision LLM, which extends the concept of any-precision DNN to LLMs by using a lightweight PTQ flow to generate a seed at $n_1$ bits and incrementally upscale to $n_K$ bits, extracting $3$ to $n$-bit variants from a single parent model. A specialized bitplane-based engine enables memory-bandwidth savings by loading only the necessary bitplanes and performing dequantization and MACs efficiently, achieving throughput that rivals or surpasses non-any-precision kernels. Across multiple models and GPUs, the approach yields state-of-the-art quality at each bit-width and substantial end-to-end performance gains, making the on-device deployment of multiple LLMs more practical and scalable. The work provides open-source code, underscoring its potential impact for developers and AI systems requiring varied latency-quality trade-offs.

Abstract

Recently, considerable efforts have been directed towards compressing Large Language Models (LLMs), which showcase groundbreaking capabilities across diverse applications but entail significant deployment costs due to their large sizes. Meanwhile, much less attention has been given to mitigating the costs associated with deploying multiple LLMs of varying sizes despite its practical significance. Thus, this paper introduces \emph{any-precision LLM}, extending the concept of any-precision DNN to LLMs. Addressing challenges in any-precision LLM, we propose a lightweight method for any-precision quantization of LLMs, leveraging a post-training quantization framework, and develop a specialized software engine for its efficient serving. As a result, our solution significantly reduces the high costs of deploying multiple, different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such as 3, 4, ..., $n$ bits, into a memory footprint comparable to a single $n$-bit LLM. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput, proving itself to be a compelling option for deployment of multiple, different-sized LLMs. Our code is open-sourced and available online.

Figures (16)

• Figure 1: Concept of any-precision quantization.
• Figure 2: Any-precision quantization via incremental upscaling.
• Figure 3: Incremental upscaling from 2 to 3-bit on non-uniform quantization methods.
• Figure 4: Comparison of (a) bitpacking-based and (b) bitplane-based representations of quantized weights.
• Figure 5: Overview of our specialized engine for any-precision LLM. We illustrate the engine running a 4-bit model while support for 3, 4 and 8-bit is available.