ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression

TL;DR

ZipLLM addresses the rapid growth of LLM storage on model hubs by co-designing model-aware deduplication and lossless compression. It introduces TensorDedup for tensor-level deduplication and BitX for lossless delta compression, augmented by a bit-distance based clustering to identify LLM families. On a large-scale survey of Hugging Face repositories, ZipLLM delivers $54.1\%$ data reduction and about $2\times$ higher throughput than state-of-the-art methods, demonstrating significant improvements in both storage and speed. The work highlights the value of leveraging model structure and lineage for storage efficiency and provides actionable guidance for deploying scalable LLM backends.

Abstract

Modern model hubs, such as Hugging Face, store tens of petabytes of LLMs, with fine-tuned variants vastly outnumbering base models and dominating storage consumption. Existing storage reduction techniques -- such as deduplication and compression -- are either LLM-oblivious or not compatible with each other, limiting data reduction effectiveness. Our large-scale characterization study across all publicly available Hugging Face LLM repositories reveals several key insights: (1) fine-tuned models within the same family exhibit highly structured, sparse parameter differences suitable for delta compression; (2) bitwise similarity enables LLM family clustering; and (3) tensor-level deduplication is better aligned with model storage workloads, achieving high data reduction with low metadata overhead. Building on these insights, we design BitX, an effective, fast, lossless delta compression algorithm that compresses XORed difference between fine-tuned and base LLMs. We build ZipLLM, a model storage reduction pipeline that unifies tensor-level deduplication and lossless BitX compression. By synergizing deduplication and compression around LLM family clustering, ZipLLM reduces model storage consumption by 54%, over 20% higher than state-of-the-art deduplication and compression approaches.

Figures (13)

• Figure 1: Left: Hugging Face's model count and storage consumption grow exponentially. Right: ZipLLM achieves both high data reduction and throughput. ZipLLM represents the end-to-end system throughput. while BitX shows the compression kernel throughput.
• Figure 2: Measurement of model repositories on Hugging Face. Note that non-LLMs contribute to a tiny fraction of storage consumption.
• Figure 3: Distribution of element-wise weight differences against Llama-3.1-8B. Top row: Deltas for three models fine-tuned directly from Llama-3.1-8B. Bottom row: Deltas for three models from the Mistral-7B-v0.3 family.
• Figure 4: Clustering of 311 LLMs by bit distance.
• Figure 5: Bitwise contribution breakdown for bit distance. Left: Bit-level differences between a fine-tuned model and its base model within the same LLM family. All models are of BF16. Right: Differences across models from different families. The Y-axis indicates the fraction of total differing bits at each bit position, computed by dividing the number of bitwise XOR results with a 1 at that position by the total number of 1s across all 16 bits.