Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens

Eugene Kwek, Wenpeng Yin

TL;DR

COMPACT addresses the inefficiency of pruning large language models by jointly pruning rare vocabulary and common-token FFN channels while preserving a standard transformer architecture. It leverages the scale-dependent parameter distribution, where embeddings dominate in small models and FFNs dominate in large models, and uses a common-token distribution to guide FFN channel importance via a weighted act$^2$ criterion. The method is training-free, architecture-agnostic, and tunable for different budgets, achieving state-of-the-art downstream performance on 0.5B–70B models with substantial reductions in parameters, GPU memory, and latency. The practical impact is improved on-device deployment, broader accessibility, and more efficient serving of LLMs across varied hardware and application settings.

Abstract

Making large language models (LLMs) more efficient in memory, latency, and serving cost is crucial for edge deployment, interactive applications, and sustainable inference at scale. Pruning is a promising technique, but existing pruning methods are limited: width pruning often breaks the standard transformer layout, requiring custom inference code, while depth pruning can cause abrupt accuracy drops. Also, while many pruning approaches are effective against LLMs, they struggle to maintain performance on small language models (SLMs). In this work, we propose COMPACT, which jointly (i) prunes rare vocabulary to shrink embedding/LM head layers and (ii) prunes FFN intermediate channels using common-token-weighted activations, aligning importance with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as: deployment-friendliness (keeps a standard transformer architecture), scale-adaptivity (trade off vocab. vs. FFN pruning), competitive pruning times, and strong memory savings alongside throughput gains. Experiments across Qwen, LLaMA, and Gemma families (0.5B-70B) show state-of-the-art downstream performance, with substantial reductions in parameters, GPU memory, and latency.

COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens

TL;DR

COMPACT addresses the inefficiency of pruning large language models by jointly pruning rare vocabulary and common-token FFN channels while preserving a standard transformer architecture. It leverages the scale-dependent parameter distribution, where embeddings dominate in small models and FFNs dominate in large models, and uses a common-token distribution to guide FFN channel importance via a weighted act criterion. The method is training-free, architecture-agnostic, and tunable for different budgets, achieving state-of-the-art downstream performance on 0.5B–70B models with substantial reductions in parameters, GPU memory, and latency. The practical impact is improved on-device deployment, broader accessibility, and more efficient serving of LLMs across varied hardware and application settings.

Abstract

Making large language models (LLMs) more efficient in memory, latency, and serving cost is crucial for edge deployment, interactive applications, and sustainable inference at scale. Pruning is a promising technique, but existing pruning methods are limited: width pruning often breaks the standard transformer layout, requiring custom inference code, while depth pruning can cause abrupt accuracy drops. Also, while many pruning approaches are effective against LLMs, they struggle to maintain performance on small language models (SLMs). In this work, we propose COMPACT, which jointly (i) prunes rare vocabulary to shrink embedding/LM head layers and (ii) prunes FFN intermediate channels using common-token-weighted activations, aligning importance with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as: deployment-friendliness (keeps a standard transformer architecture), scale-adaptivity (trade off vocab. vs. FFN pruning), competitive pruning times, and strong memory savings alongside throughput gains. Experiments across Qwen, LLaMA, and Gemma families (0.5B-70B) show state-of-the-art downstream performance, with substantial reductions in parameters, GPU memory, and latency.

Paper Structure

This paper contains 32 sections, 5 equations, 5 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vocabulary Size/Pruning.
  4. Proposed Method: Compact
  5. Analyzing Parameter Distribution in LLMs: Vocabulary vs. FFN vs. Attention
  6. From Rare to Common: Rationale of Vocabulary Pruning
  7. Intermediate Pruning under the Common-Token Distribution
  8. Compact: Joint Pruning Pipeline
  9. Experiments
  10. Experimental Setup
  11. Performance on downstream tasks
  12. Results on Smaller LLMs
  13. Compact outperforms baselines by large margins.
  14. Compact supports a wide variety of model architectures out-of-the-box.
  15. Results on larger LLMs
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Parameter distribution across Qwen 2.5 models of different scales.
  • Figure 2: Common-act$^2$ outperforms both act$^2$ and $|$act$|$ (Qwen 2.5-0.5B at a 35% pruning ratio).
  • Figure 3: Downstream performance by calibration dataset size. Although all benchmarks used 256 calibration samples, 16 samples is sufficient.
  • Figure 4: Llama 3 Parameter Distribution
  • Figure 5: Gemma 3 Parameter Distribution