Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightweight error mitigation strategies for post-training N:M activation sparsity in LLMs

Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina, Vladimir Smirnov, Redko Dmitry, Alexey Dontsov, Maxim Zhelnin, Evgeny Burnaev, Egor Shvetsov

TL;DR

This work presents a comprehensive analysis of methods for post-training N:M activation pruning in LLMs, demonstrating that pruning activations enables superior preservation of generative capabilities compared to weight pruning at equivalent sparsity levels.

Abstract

The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for weights, its application to activation pruning remains underexplored despite its potential for dynamic, input-adaptive compression and reductions in I/O overhead. This work presents a comprehensive analysis of methods for post-training N:M activation pruning in LLMs. Across multiple LLMs, we demonstrate that pruning activations enables superior preservation of generative capabilities compared to weight pruning at equivalent sparsity levels. We evaluate lightweight, plug-and-play error mitigation techniques and pruning criteria, establishing strong hardware-friendly baselines that require minimal calibration. Furthermore, we explore sparsity patterns beyond NVIDIA's standard 2:4, showing that the 16:32 pattern achieves performance nearly on par with unstructured sparsity. However, considering the trade-off between flexibility and hardware implementation complexity, we focus on the 8:16 pattern as a superior candidate. Our findings provide both effective practical methods for activation pruning and a motivation for future hardware to support more flexible sparsity patterns. Our code is available https://anonymous.4open.science/r/Structured-Sparse-Activations-Inference-EC3C/README.md .

Lightweight error mitigation strategies for post-training N:M activation sparsity in LLMs

TL;DR

This work presents a comprehensive analysis of methods for post-training N:M activation pruning in LLMs, demonstrating that pruning activations enables superior preservation of generative capabilities compared to weight pruning at equivalent sparsity levels.

Abstract

The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for weights, its application to activation pruning remains underexplored despite its potential for dynamic, input-adaptive compression and reductions in I/O overhead. This work presents a comprehensive analysis of methods for post-training N:M activation pruning in LLMs. Across multiple LLMs, we demonstrate that pruning activations enables superior preservation of generative capabilities compared to weight pruning at equivalent sparsity levels. We evaluate lightweight, plug-and-play error mitigation techniques and pruning criteria, establishing strong hardware-friendly baselines that require minimal calibration. Furthermore, we explore sparsity patterns beyond NVIDIA's standard 2:4, showing that the 16:32 pattern achieves performance nearly on par with unstructured sparsity. However, considering the trade-off between flexibility and hardware implementation complexity, we focus on the 8:16 pattern as a superior candidate. Our findings provide both effective practical methods for activation pruning and a motivation for future hardware to support more flexible sparsity patterns. Our code is available https://anonymous.4open.science/r/Structured-Sparse-Activations-Inference-EC3C/README.md .

Paper Structure

This paper contains 26 sections, 13 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Our Contributions:
  3. Methods
  4. Preliminaries
  5. Pruning Criterion
  6. Transformations for Error Mitigation
  7. Evaluation & Models
  8. Results
  9. Sparse Weights vs. Activations
  10. Optimal semi-structured sparsity patterns
  11. Results on Single/Multi-choice Datasets
  12. Evaluation of Pruning Criteria
  13. Evaluation of Transformations
  14. Instruction-Following Tasks
  15. Results for Unstructured Pruning
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Comparison of unstructuredsparsity in activations (ACT) and weights (WT) averaged across four datasets at varying sparsity ratios. Higher is Better. More detailed results are presented in Appendix Table \ref{['tab:unstructured_pruning']}.
  • Figure 2: Comparison of sparsity patterns with unstructured sparsity. 50% and 70% correspond to unstructured sparsity. Lower is Better. More detailed results are presented in Appendix Table \ref{['tab:sparsity_patterns']}.