Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MI-PRUN: Optimize Large Language Model Pruning via Mutual Information

Hao Zhang, Zhibin Zhang, Guangxin Wu, He Chen, Jiafeng Guo, Xueqi Cheng

TL;DR

This paper proposes a mutual information based pruning method MI-PRUN for LLMs that leverages mutual information to identify redundant blocks by evaluating transitions in hidden states and develops the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency.

Abstract

Large Language Models (LLMs) have become indispensable across various domains, but this comes at the cost of substantial computational and memory resources. Model pruning addresses this by removing redundant components from models. In particular, block pruning can achieve significant compression and inference acceleration. However, existing block pruning methods are often unstable and struggle to attain globally optimal solutions. In this paper, we propose a mutual information based pruning method MI-PRUN for LLMs. Specifically, we leverages mutual information to identify redundant blocks by evaluating transitions in hidden states. Additionally, we incorporate the Data Processing Inequality (DPI) to reveal the relationship between the importance of entire contiguous blocks and that of individual blocks. Moreover, we develop the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency. Extensive experiments across various models and datasets demonstrate the stability and effectiveness of our method.

MI-PRUN: Optimize Large Language Model Pruning via Mutual Information

TL;DR

This paper proposes a mutual information based pruning method MI-PRUN for LLMs that leverages mutual information to identify redundant blocks by evaluating transitions in hidden states and develops the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency.

Abstract

Large Language Models (LLMs) have become indispensable across various domains, but this comes at the cost of substantial computational and memory resources. Model pruning addresses this by removing redundant components from models. In particular, block pruning can achieve significant compression and inference acceleration. However, existing block pruning methods are often unstable and struggle to attain globally optimal solutions. In this paper, we propose a mutual information based pruning method MI-PRUN for LLMs. Specifically, we leverages mutual information to identify redundant blocks by evaluating transitions in hidden states. Additionally, we incorporate the Data Processing Inequality (DPI) to reveal the relationship between the importance of entire contiguous blocks and that of individual blocks. Moreover, we develop the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency. Extensive experiments across various models and datasets demonstrate the stability and effectiveness of our method.
Paper Structure (12 sections, 6 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 12 sections, 6 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Block Pruning
  4. Mutual Information
  5. Method
  6. Mutual Information Measures Block Importance
  7. Importance of Continuous Blocks
  8. Fast-Block-Select (Iterative Updates)
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Conclusion

Figures (4)

  • Figure 1: The relationship between the mutual information of the global continuous block and the local individual blocks.
  • Figure 2: An overview of our method. The process of pruning 5 blocks in the LLaMA2-7B model. We provide a detailed description of the implementation for each step in Section \ref{['Fast-Block-Select']}.
  • Figure 3: Performance of Llama2-7B with increasing pruning blocks.
  • Figure 4: Performance comparison between Greedy Strategy and Iterative Update.