MoDeGPT: Modular Decomposition for Large Language Model Compression

TL;DR

MoDeGPT tackles the challenge of deploying large language models on resource-constrained devices by introducing a training-free, modular decomposition framework. By grouping weight matrices into modules and applying Nyström, CR, or SVD to module-level matrix pairs, it achieves high compression with minimal loss in zero-shot performance and substantial throughput gains. The approach is underpinned by a formal modular reconstruction objective and a global sparsity allocation strategy, with strong theoretical bounds on reconstruction error. Empirically, MoDeGPT delivers competitive or superior results to gradient-based and non-gradient baselines, while significantly reducing compute and improving inference speed, enabling practical deployment of large models like LLaMA-2/3 and OPT at substantial scale.

Abstract

Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nyström approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%.

Figures (16)

• Figure 1: Comparison of Matrix Decomposition-Based Methods for Transformer Compression.(a) Original transformer layer. (b) SVD applied to each weight matrix separately, resulting in dual matrices. (c) SliceGPT multiplies each weight matrix by an orthogonal matrix ${\bm{Q}}$, reducing dimensions and introducing additional adapters. (d) MoDeGPT organizes matrices into modules (highlighted by green boxes) and jointly decomposes them, producing reduced-size matrices without extra adapters.
• Figure 2: The MoDeGPT Framework. MoDeGPT divides a transformer layer into three distinct colored modules, each optimizing two matrices using a specific low-rank approximation method. A twill hatch pattern represents the dimension reduction.
• Figure 3: Module characteristics and their associated matrix decompositions.
• Figure 4: Compute time.
• Figure 6: Module-wise compression.

Theorems & Definitions (16)

• Definition 1
• Theorem 1: MLP compression by Nyström approximation
• Proposition 1: MLP compression error
• Theorem 2: Key-Query compression by CR approximation
• Proposition 2: Key-Query compression error
• Theorem 3: Value-Output compression by SVD
• Proposition 3: Value-Output compression error
• Theorem 4
• Definition 2: Column Selection Matrix
• proof