Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference

Sihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury, Huiping Zhuang, Luming Liang, Tianyi Chen

TL;DR

WINA addresses the growing inference cost of large language models by proposing a training-free sparse-activation scheme that jointly leverages hidden-state magnitudes and column-wise $\ell_2$ weight norms to select influential neurons. The method provides theoretical guarantees showing tighter approximation-error bounds than prior approaches like TEAL, and it adapts to real models through a tensor-transformation step that enforces column-orthogonality while preserving output. Empirically, WINA achieves higher accuracy than state-of-the-art training-free baselines across multiple 7B–14B LLMs and datasets, with substantial GFLOP reductions (up to about 60–63% at 65% sparsity). Overall, WINA offers a practical, plug-and-play path to efficient LLM inference with strong theoretical and empirical support.

Abstract

The growing computational demands of large language models (LLMs) make efficient inference and activation strategies increasingly critical. While recent approaches, such as Mixture-of-Experts (MoE), leverage selective activation but require specialized training, training-free sparse activation methods offer broader applicability and superior resource efficiency through their plug-and-play design. However, many existing methods rely solely on hidden state magnitudes to determine activation, resulting in high approximation errors and suboptimal inference accuracy. To address these limitations, we propose WINA (Weight Informed Neuron Activation), a novel, simple, and training-free sparse activation framework that jointly considers hidden state magnitudes and the column-wise $\ell_2$-norms of weight matrices. We show that this leads to a sparsification strategy that obtains optimal approximation error bounds with theoretical guarantees tighter than existing techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g., TEAL) by up to $2.94\%$ in average performance at the same sparsity levels, across a diverse set of LLM architectures and datasets. These results position WINA as a new performance frontier for training-free sparse activation in LLM inference, advancing training-free sparse activation methods and setting a robust baseline for efficient inference. The source code is available at https://github.com/microsoft/wina.

WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference

TL;DR

WINA addresses the growing inference cost of large language models by proposing a training-free sparse-activation scheme that jointly leverages hidden-state magnitudes and column-wise weight norms to select influential neurons. The method provides theoretical guarantees showing tighter approximation-error bounds than prior approaches like TEAL, and it adapts to real models through a tensor-transformation step that enforces column-orthogonality while preserving output. Empirically, WINA achieves higher accuracy than state-of-the-art training-free baselines across multiple 7B–14B LLMs and datasets, with substantial GFLOP reductions (up to about 60–63% at 65% sparsity). Overall, WINA offers a practical, plug-and-play path to efficient LLM inference with strong theoretical and empirical support.

Abstract

The growing computational demands of large language models (LLMs) make efficient inference and activation strategies increasingly critical. While recent approaches, such as Mixture-of-Experts (MoE), leverage selective activation but require specialized training, training-free sparse activation methods offer broader applicability and superior resource efficiency through their plug-and-play design. However, many existing methods rely solely on hidden state magnitudes to determine activation, resulting in high approximation errors and suboptimal inference accuracy. To address these limitations, we propose WINA (Weight Informed Neuron Activation), a novel, simple, and training-free sparse activation framework that jointly considers hidden state magnitudes and the column-wise -norms of weight matrices. We show that this leads to a sparsification strategy that obtains optimal approximation error bounds with theoretical guarantees tighter than existing techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g., TEAL) by up to in average performance at the same sparsity levels, across a diverse set of LLM architectures and datasets. These results position WINA as a new performance frontier for training-free sparse activation in LLM inference, advancing training-free sparse activation methods and setting a robust baseline for efficient inference. The source code is available at https://github.com/microsoft/wina.

Paper Structure

This paper contains 39 sections, 4 theorems, 53 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Sparse Activation.
  5. Relations to Model Structured Pruning.
  6. Methodology
  7. Problem Statement
  8. Weight Informed Gate Function
  9. Theoretical Analysis
  10. Proof.
  11. Proof.
  12. Proof.
  13. Proof.
  14. Remark.
  15. From Theory to Practice
  16. ...and 24 more sections

Key Result

Lemma 3.1

Let $\bm{x} \in \mathbb{R}^n$ be an input vector and $W \in \mathbb{R}^{m \times n}$ be a matrix satisfying column-wise orthogonality: $W^\top W= I_{n}$ where $I_n$ is an identity matrix. For any target sparsity level $k \in \mathbb{N}^+$ satisfying $k < n$, the expected deviation between the origin where $\bm{x}_\text{WINA{}{}}$ is the sparse input via WINA, retaining the $k$ elements activated w

Figures (2)

  • Figure 1: Overview of WINA. WINA performs training-free sparse activation by selecting the most influential input dimensions based on both hidden state magnitudes and the column-wise $\ell_2$-norms of weight matrices. This joint criterion ensures accurate sub-network activation at each layer during inference, preserving model performance while reducing computational overhead.
  • Figure 2: Sparsity-performance frontiers. Sparsity-performance across Qwen-2.5-7B, Llama-2-7B, Llama-3-8B, and Phi-4-14B.

Theorems & Definitions (5)

  • Lemma 3.1: Optimal approximation error over single layer
  • Theorem 3.2: Optimal approximation error over consecutive $L$ layer
  • Definition 3.3: Monotonic increasing function (MIF)
  • Lemma 3.4: Optimal approximation error over a single layer with MIF
  • Theorem 3.5: Optimal approximation error over consecutive $L$ layer with MIF