E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity

TL;DR

E-Sparse introduces an entropy-based one-shot N:M sparsity for large language models, leveraging channel-wise information richness and a channel-shuffle mechanism to preserve information-rich channels under pruning without weight updates. The method combines a novel entropy-based channel importance metric with global naive and local block channel reordering, implemented as Sparse-GEMM in FasterTransformer and accelerated by cuSPARSE/cuSPARSELt on NVIDIA Ampere GPUs. Across LLaMA and OPT, E-Sparse delivers significant inference speedups (up to 1.53x) and memory savings (up to 43.5%) with negligible accuracy loss, outperforming prior training-free sparsity methods. Ablations confirm the benefits of entropy, Global Naive Shuffle, and Local Block Shuffle, and results generalize across model sizes and zero-shot tasks.

Abstract

Traditional pruning methods are known to be challenging to work in Large Language Models (LLMs) for Generative AI because of their unaffordable training process and large computational demands. For the first time, we introduce the information entropy of hidden state features into a pruning metric design, namely E-Sparse, to improve the accuracy of N:M sparsity on LLM. E-Sparse employs the information richness to leverage the channel importance, and further incorporates several novel techniques to put it into effect: (1) it introduces information entropy to enhance the significance of parameter weights and input feature norms as a novel pruning metric, and performs N:M sparsity without modifying the remaining weights. (2) it designs global naive shuffle and local block shuffle to quickly optimize the information distribution and adequately cope with the impact of N:M sparsity on LLMs' accuracy. E-Sparse is implemented as a Sparse-GEMM on FasterTransformer and runs on NVIDIA Ampere GPUs. Extensive experiments on the LLaMA family and OPT models show that E-Sparse can significantly speed up the model inference over the dense model (up to 1.53X) and obtain significant memory saving (up to 43.52%), with acceptable accuracy loss.

Figures (3)

• Figure 1: Overview of the proposed E-Sparse. It first introduces entropy to quantify the information richness within each channel ( intra-channel ) of the input features, and adopts it to enhance the feature norms ( cross-channel ) as a metric to evaluate parameter importance. Furthermore, it proposes Channel Shuffle to reorder the information distribution in LLMs to obtain N:M Sparsity with less information loss.
• Figure 2: The visualization of the hidden activations in LLMs. The data for each subfigure comes from the activation of the corresponding layer of LLaMA-13B. For clarity, we only capture the norm and entropy values for the 100 channels after norm sorting in (a) and (b). We show the entropy values of all channels in (c) and (d).
• Figure 3: Channel Shuffle of E-Sparse. Take 2:4 sparsity as an example. E-Sparse first sorts the channels globally according to the channel mean of the sparsity metric, and then divides the channels with close mean into different groups, which is coarse-grained but faster. Then, E-Sparse splits the channel into multiple blocks and performs channel shuffle within the blocks, which is slightly slower than the global shuffling but more accurate.