Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PocketLLM: Ultimate Compression of Large Language Models via Meta Networks

Ye Tian, Chengcheng Wang, Jing Han, Yehui Tang, Kai Han

TL;DR

PocketLLM tackles the challenge of edge deployment for ever-larger LLMs by compressing weights in a latent space using meta-encoders/decoders and a compact codebook. The approach encodes weight subvectors into discrete latent representations, replaces them with codebook indices, and reconstructs the original weights with a meta decoder, requiring only the decoder, codebook, and indices to be stored. Empirical results show state-of-the-art performance at extreme compression ratios (e.g., up to $20\times$) with optional fine-tuning restoring dense-model accuracy, highlighting the method’s practicality for edge devices and low-bandwidth environments. This latent-space compression reduces storage and transmission costs while maintaining strong zero-shot performance across multiple benchmarks, signaling a substantial advancement in scalable LLM deployment. The work introduces robust components like Reshaped Layer Normalization and a straight-through estimator for non-differentiable codebook lookups, contributing broadly to latent-variable quantization techniques in large-scale models.

Abstract

As Large Language Models (LLMs) continue to grow in size, storing and transmitting them on edge devices becomes increasingly challenging. Traditional methods like quantization and pruning struggle to achieve extreme compression of LLMs without sacrificing accuracy. In this paper, we introduce PocketLLM, a novel approach to compress LLMs in a latent space via meta-networks. A simple encoder network is proposed to project the weights of LLMs into discrete latent vectors, which are then represented using a compact codebook. A lightweight decoder network is employed to map the codebook's representative vectors back to the original weight space. This method allows for significant compression of the large weights in LLMs, consisting solely of a small decoder, a concise codebook, and an index. Extensive experiments show that PocketLLM achieves superior performance even at significantly high compression ratios, e.g., compressing Llama 2-7B by 10x with a negligible drop in accuracy.

PocketLLM: Ultimate Compression of Large Language Models via Meta Networks

TL;DR

PocketLLM tackles the challenge of edge deployment for ever-larger LLMs by compressing weights in a latent space using meta-encoders/decoders and a compact codebook. The approach encodes weight subvectors into discrete latent representations, replaces them with codebook indices, and reconstructs the original weights with a meta decoder, requiring only the decoder, codebook, and indices to be stored. Empirical results show state-of-the-art performance at extreme compression ratios (e.g., up to ) with optional fine-tuning restoring dense-model accuracy, highlighting the method’s practicality for edge devices and low-bandwidth environments. This latent-space compression reduces storage and transmission costs while maintaining strong zero-shot performance across multiple benchmarks, signaling a substantial advancement in scalable LLM deployment. The work introduces robust components like Reshaped Layer Normalization and a straight-through estimator for non-differentiable codebook lookups, contributing broadly to latent-variable quantization techniques in large-scale models.

Abstract

As Large Language Models (LLMs) continue to grow in size, storing and transmitting them on edge devices becomes increasingly challenging. Traditional methods like quantization and pruning struggle to achieve extreme compression of LLMs without sacrificing accuracy. In this paper, we introduce PocketLLM, a novel approach to compress LLMs in a latent space via meta-networks. A simple encoder network is proposed to project the weights of LLMs into discrete latent vectors, which are then represented using a compact codebook. A lightweight decoder network is employed to map the codebook's representative vectors back to the original weight space. This method allows for significant compression of the large weights in LLMs, consisting solely of a small decoder, a concise codebook, and an index. Extensive experiments show that PocketLLM achieves superior performance even at significantly high compression ratios, e.g., compressing Llama 2-7B by 10x with a negligible drop in accuracy.

Paper Structure

This paper contains 18 sections, 13 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Approach
  4. Prelimineries
  5. Transformer
  6. PocketLLM Framework
  7. Encoding Weights into Latent Spaces
  8. Reshaped Layer Normalization
  9. Compression in Latent Spaces
  10. Weight Reconstruction with Meta Decoders
  11. Compression Ratio Analysis
  12. Algorithm Process Overview
  13. Experiments
  14. Implement Details
  15. Comparisons with State-of-the-art Methods
  16. ...and 3 more sections

Figures (3)

  • Figure 1: PocketLLM framework: compressing weight vectors in latent spaces with meta-networks.
  • Figure 2: We visualize the numerical distribution of values within the 99.9% range in the value projection weights $W_v$.
  • Figure 3: The reconstructed results of $q$, $up$, $down$ weight vectors correspond to the compression ratios of $8\times$ (4 bits), $16\times$ (2 bits), and $20\times$ (1.6 bits), respectively.