QStore: Quantization-Aware Compressed Model Storage

TL;DR

QStore addresses the practical problem of storing multiple precision variants of large foundation models by proposing a unified lossless format that stores a low-precision base together with residual information required to reconstruct high-precision weights. Grounded in information theory, it models the problem as $H(W)=H(Q(W))+H(W\mid Q(W))$ and uses joint compression with grouping by quantized values and quantization functions to achieve high compression ratios while enabling fast, lossless decoding on CPU. Empirically, QStore achieves up to 2.2x storage savings and up to 2x faster loading/saving across six foundation models, and generalizes to multiple precisions and datatypes, including multi-level model chains. The approach supports lazy loading, pipelined decoding, and compatibility with common model formats, offering a practical path to reduce storage costs and improve deployment efficiency in model hubs and downstream applications.

Abstract

Modern applications commonly leverage large, multi-modal foundation models. These applications often feature complex workflows that demand the storage and usage of similar models in multiple precisions. A straightforward approach is to maintain a separate file for each model precision (e.g., INT8, BF16), which is indeed the approach taken by many model providers such as HuggingFace and Ollama. However, this approach incurs excessive storage costs since a higher precision model (e.g., BF16) is a strict superset of a lower precision model (e.g., INT8) in terms of information. Unfortunately, simply maintaining only the higher-precision model and requiring every user to dynamically convert the model precision is not desirable because every user of lower precision models must pay the cost for model download and precision conversion. In this paper, we present QStore, a unified, lossless compression format for simultaneously storing a model in two (high and low) precisions efficiently. Instead of storing low-precision and high-precision models separately, QStore stores low-precision model and only the residual information needed to reconstruct high-precision models. The size of residual information is significantly smaller than the original high-precision models, thus achieving high savings in storage cost. Moreover, QStore does not compromise the speed of model loading. The low-precision models can be loaded quickly just like before. The high-precision models can also be reconstructed efficiently in memory by merging low-precision data and the residual with QStore's lightweight decoding logic. We evaluate QStore for compressing multiple precisions of popular foundation models, and show that QStore reduces overall storage footprint by up to 2.2x (45% of the original size) while enabling up to 1.7x and 1.8x faster model saving and loading versus existing approaches.

Figures (10)

• Figure 1: QStore stores the conditional bit representation of the high-precision model weights alongside the common low-precision, quantized weights to store a model in both high and low-precisions using fewer bits.
• Figure 2: QStore pipeline. A high and low-precision model pair is encoded into a unified, storage-friendly format ( QStore), from which both models can be efficiently retrieved.
• Figure 3: Weighted entropy of different grouping strategies on the Llama 3.1 8B Instruct model's 16-bit weights. QStore's combined grouping achieves high entropy reduction (hence compression ratio) versus alternative grouping strategies.
• Figure 4: Compressing a tensor in the model pair with QStore. Weights in the high-precision model are grouped by the quantization function (scale) applied, then subgrouped by the post-quantization value from the low-precision model.
• Figure 5: Decoding time when we need both high-precision and low-precision models, versus storage costs: QStore's loads the model pair up to $2.2\times$ and $1.8\times$ faster versus loading uncompressed models and compression baselines.