ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models

TL;DR

Big language models incur heavy inference costs from dense activations. The authors re-evaluate ReLU activations, showing highly sparse FFN activations (>90%) and substantial inference and memory savings across OPT, Llama, and Falcon, including up to about threefold FLOPS reductions through Relufication. They introduce Stage 1 FFN replacement and Stage 2 post-normalization ReLU insertions to boost sparsity with minimal accuracy loss, and they explore applications such as aggregated sparsity for token generation, speculative decoding improvements, and shifted ReLU to further enhance sparsity. Overall, activation sparsity via ReLU presents a practical, hardware-friendly path to more efficient LLM deployment, with strong empirical support and clear avenues for future optimization.

Abstract

Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained devices. Despite recent trends favoring alternative activation functions such as GELU or SiLU, known for increased computation, this study strongly advocates for reinstating ReLU activation in LLMs. We demonstrate that using the ReLU activation function has a negligible impact on convergence and performance while significantly reducing computation and weight transfer. This reduction is particularly valuable during the memory-bound inference step, where efficiency is paramount. Exploring sparsity patterns in ReLU-based LLMs, we unveil the reutilization of activated neurons for generating new tokens and leveraging these insights, we propose practical strategies to substantially reduce LLM inference computation up to three times, using ReLU activations with minimal performance trade-offs.

Figures (12)

• Figure 1: (a) Activation Sparsity of different pretrained models: ReLU-based OPTs show significantly higher sparsity. (b) Zeroed out entries after ReLU save compute in large semi-structured chunks (e.g., rows). (c) Comparison of inference efficiency and performance of the different models with different activation functions after fine-tuning: The choice of activation function does not significantly impact the accuracy, as any of GELU, SiLU, or ReLU can be used on all three models and achieve the same level of accuracy as the original activation function. However, using ReLU can provide an additional benefit of leading to activation sparsity and faster inference.
• Figure 2: (top) (a) Shapes of different gating functions over [-5, 5]; (b) Continuation of (a) where SiLU is comparably larger compared to others; (c) Sparsity of the FFN with different activations: increasing $\beta$ increases sparsity. (bottom) when trained from scratch, OPT 1.3 B models using different activation functions achieve similar performance.
• Figure 3: Architectural surgeries for relufication. In stage 1 we keep the existing ReLUs (in the case of OPT) or replace the activation function between up projection and down projections from GELU (Falcon) and SiLU (Llama) to ReLU. In stage 2, we insert new ReLUs after normalization layers.
• Figure 4: Activation sparsity of Falcon and Llama models improves significantly after relufication.
• Figure 5: The preactivation distribution of pretrained models for Falcon and Llama does not change significantly during the short finetuning stage of relufication. The dashed line shows the cutoff point before which the output is almost zero.

Theorems & Definitions (4)

• Theorem 1
• proof
• Theorem 2
• proof