Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models

Zhiyu Guo, Hidetaka Kamigaito, Taro Wanatnabe

TL;DR

Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent GLU-based LLMs pruning, which incorporates structural dependency into the weight magnitude-based unstructured pruning, is proposed.

Abstract

The rapid advancement in Large Language Models (LLMs) has markedly enhanced the capabilities of language understanding and generation. However, the substantial model size poses hardware challenges, affecting both memory size for serving and inference latency for token generation. To address those challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent GLU-based LLMs pruning, which incorporates structural dependency into the weight magnitude-based unstructured pruning. We introduce an MLP-specific pruning metric that evaluates the importance of each weight by jointly considering its magnitude and its corresponding MLP intermediate activation norms. DaSS facilitates a balance between the adaptability offered by unstructured pruning and the structural consistency inherent in dependency-based structured pruning. Empirical evaluations on LLaMA2, Mistral, and Gemma model families demonstrate that DaSS not only outperforms both SparseGPT and Wanda in achieving hardware-friendly N:M sparsity patterns but also maintains the computational efficiency of Wanda.

Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models

TL;DR

Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent GLU-based LLMs pruning, which incorporates structural dependency into the weight magnitude-based unstructured pruning, is proposed.

Abstract

The rapid advancement in Large Language Models (LLMs) has markedly enhanced the capabilities of language understanding and generation. However, the substantial model size poses hardware challenges, affecting both memory size for serving and inference latency for token generation. To address those challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent GLU-based LLMs pruning, which incorporates structural dependency into the weight magnitude-based unstructured pruning. We introduce an MLP-specific pruning metric that evaluates the importance of each weight by jointly considering its magnitude and its corresponding MLP intermediate activation norms. DaSS facilitates a balance between the adaptability offered by unstructured pruning and the structural consistency inherent in dependency-based structured pruning. Empirical evaluations on LLaMA2, Mistral, and Gemma model families demonstrate that DaSS not only outperforms both SparseGPT and Wanda in achieving hardware-friendly N:M sparsity patterns but also maintains the computational efficiency of Wanda.
Paper Structure (33 sections, 5 equations, 2 figures, 10 tables)

This paper contains 33 sections, 5 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Wanda Pruning Method
  4. GLU Variants for Transformer
  5. Output-balanced Pruning Limitations
  6. Dependency-aware Semi-structured Sparsity
  7. Structure Dependency in MLP.
  8. Incorporating Dependency into Weight Importance.
  9. Pruning Granularity.
  10. Extension to N:M Sparsity.
  11. Discussion.
  12. Experiments
  13. Settings
  14. Models.
  15. Baseline Approaches.
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Illustration of Dependency-aware Semi-structured Sparsity (DaSS). In (a) dependency-based structured pruning ma2023llm, all the weights connecting to the same intermediate neuron are removed or remain simultaneously. In (b) Wanda unstructured pruning sun2024a, it assigns greater emphasis to the weights corresponding to large input activations. For Gate-Proj and UP-Proj, the same number of weights are removed for each MLP neuron, regardless of whether some neurons have much larger activation norms. For Down-Proj, the weights corresponding to larger activation norms are more likely to be pruned. This can lead to a structural mismatch. In (c), all the weights corresponding to large intermediate activations are more likely to be reserved.
  • Figure 2: Robustness to calibration samples.