Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Search for Efficient Large Language Models

Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, Yanzhi Wang

TL;DR

A training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration, and introduces a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data.

Abstract

Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs. In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data. Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration. Code: https://github.com/shawnricecake/search-llm

Search for Efficient Large Language Models

TL;DR

A training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration, and introduces a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data.

Abstract

Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs. In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data. Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration. Code: https://github.com/shawnricecake/search-llm
Paper Structure (25 sections, 1 theorem, 16 equations, 10 figures, 10 tables, 1 algorithm)

This paper contains 25 sections, 1 theorem, 16 equations, 10 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Compression of LLMs
  4. Search for Transformers
  5. Methodology
  6. Framework Overview
  7. Search Initialization
  8. Architecture Search
  9. Mask Mutation
  10. Search Space
  11. Search Pipeline
  12. Reformation
  13. Efficient Inference
  14. Experiments
  15. Experiment Settings
  16. ...and 10 more sections

Key Result

Theorem 3.1

Problem (eq:ori_problem) can be solved in iterations. In the $k^{th}$ iteration, it performs the updates: where $\rho > 0$ is the penalty parameter. The initial variable values at $k=0$ follow the configurations that $\widehat{ \mathbf W}^0 = \mathbf W$, $\mathbf Z^0 = \widehat{ \mathbf W}^0$, and $\mathbf U^{0} = \mathbf 0$.

Figures (10)

  • Figure 1: Experiment results of perplexity $\downarrow$ on WikiText2 dataset with 2048 sequence length.
  • Figure 2: Framework Overview.
  • Figure 3: Visualization of the subnets generation for LLaMA family based on the selections masks $\mathbf{S}_{attn}$ for the self-attention module colored in blue and $\mathbf{S}_{mlp}$ for the MLP module colored in green.
  • Figure 4: Ablation analysis of the inheriting ratios applied to the self-attention, MLP, or both.
  • Figure 5: Ablation analysis of convergence speed with or without our initialization.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Theorem 3.1