SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models

TL;DR

SEAP tackles the high inference cost of large language models by introducing a training-free, task-adaptive pruning framework guided by task-specific activation patterns. It computes neuron importance scores $s_i^{(\ell,\tau)}$ from activation statistics and weights, then distributes sparsity across layers via a logistic map with target $G$, enabling Expert-Based or General pruning. Empirically, SEAP on Llama-2-7B/13B achieves substantial speedups with minimal accuracy loss (e.g., ~2.2% at 20% pruning and >20% improvement over baselines at 50% pruning), demonstrating scalable, deployment-ready efficiency improvements. These findings indicate that task-aware activation pruning can meaningfully reduce compute and memory demands for large-scale LLMs while preserving task performance, facilitating real-world, resource-constrained deployment.

Abstract

Large Language Models have achieved remarkable success across various natural language processing tasks, yet their high computational cost during inference remains a major bottleneck. This paper introduces Sparse Expert Activation Pruning (SEAP), a training-free pruning method that selectively retains task-relevant parameters to reduce inference overhead. Inspired by the clustering patterns of hidden states and activations in LLMs, SEAP identifies task-specific expert activation patterns and prunes the model while preserving task performance and enhancing computational efficiency. Experimental results demonstrate that SEAP significantly reduces computational overhead while maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2% performance drop compared to the dense model. These findings highlight SEAP's scalability and effectiveness, making it a promising approach for optimizing large-scale LLMs.

Figures (7)

• Figure 1: Visualization of hidden states $h(P)$ from different tasks. Each point represents the activation of a hidden state in the model for a specific task. The clustering patterns illustrate how tasks with similar requirements tend to activate similar regions in the model.
• Figure 2: Framework of the SEAP approach. The left side shows the Motivation Discovery phase, where task-specific activation patterns are identified by analyzing hidden states and neuron activations extracted from the task corpus. The right side illustrates the Training-free Sparse Expert Activation Pruning process, consisting of five main steps described in Section \ref{['subsec: SEAD']}.
• Figure 3: Heatmaps of dimension-wise average normalized $\ell$2 norms for different tasks. Each row corresponds to a layer or module, and each column represents a dimension in the hidden state space. The top and bottom parts of the figure show activation patterns from two randomly selected subsets of the same task. Consistent color patterns appear within tasks of the same type, while distinctly different tasks exhibit unique activation signatures, supporting our hypothesis that tasks selectively activate specific dimensions.
• Figure 4: Illustration of how neurons are pruned based on importance scores.
• Figure 5: Impact of pruning on MMLU performance at different layers and sparsity levels. Early layers are more sensitive to pruning.