Activation Sparsity Opportunities for Compressing General Large Language Models
Nobel Dhar, Bobin Deng, Md Romyull Islam, Kazi Fahim Ahmad Nasif, Liang Zhao, Kun Suo
TL;DR
This work targets the deployment of large language models on edge devices by investigating activation sparsity as a compression lever that is orthogonal to pruning and quantization. By focusing on FFN layers, which comprise about \frac{2}{3} of LLM parameters, the authors enforce activation sparsity through percentile-based thresholds without retraining, achieving about \$50\%$ extra sparsity with negligible perplexity degradation. They show that activation patterns are highly predictable, enabling prefetching of only the active weights and discarding inactive ones to reduce memory and latency on resource-constrained hardware. The results are demonstrated across diverse non-ReLU LLMs and are supported by a practical analysis toolkit; the proposed predictor-guided approach can be integrated with existing compression pipelines and extended to other transformer-based models, potentially delivering substantial edge deployment benefits.
Abstract
Deploying local AI models, such as Large Language Models (LLMs), to edge devices can substantially enhance devices' independent capabilities, alleviate the server's burden, and lower the response time. Owing to these tremendous potentials, many big tech companies have released several lightweight Small Language Models (SLMs) to bridge this gap. However, we still have huge motivations to deploy more powerful (LLMs) AI models on edge devices and enhance their smartness level. Unlike the conventional approaches for AI model compression, we investigate activation sparsity. The activation sparsity method is orthogonal and combinable with existing techniques to maximize the compression rate while maintaining great accuracy. LLMs' Feed-Forward Network (FFN) components, which typically comprise a large proportion of parameters (around 2/3), ensure that our FFN optimizations would have a better chance of achieving effective compression. Moreover, our findings are beneficial to general LLMs and are not restricted to ReLU-based models. This work systematically investigates the tradeoff between enforcing activation sparsity and perplexity (accuracy) on state-of-the-art LLMs. Our empirical analysis demonstrates that we can obtain around 50% of main memory and computing reductions for critical FFN components with negligible accuracy degradation. This extra 50% sparsity does not naturally exist in the current LLMs, which require tuning LLMs' activation outputs by injecting zero-enforcing thresholds. To obtain the benefits of activation sparsity, we provide a guideline for the system architect for LLM prediction and prefetching. The success prediction allows the system to prefetch the necessary weights while omitting the inactive ones and their successors, therefore lowering cache and memory pollution and reducing LLM execution time on resource-constrained edge devices.