Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Dynamic Sparsity in Quantized LLM Inference

Rongxiang Wang, Kangyuan Shu, Felix Xiaozhu Lin

TL;DR

This paper addresses the challenge of running quantized LLMs on client devices by leveraging dynamic activation sparsity without sacrificing the benefits of low-bit quantization. It introduces SpQt, a co-design consisting of a Zigzag weight layout aligned with activation sparsity and a sparsity-aware Zigzag GEMV kernel accompanied by a runtime sparse index collector. The key contributions are the Zigzag layout, the specialized kernels and index-collection mechanism, and an end-to-end system validated on Apple GPUs across Llama-2/3 models, achieving up to 1.55× end-to-end speedups and up to 1.8× GFLOPS improvements with minimal accuracy loss. This work demonstrates that structured sparsity and quantization can coexist effectively on commodity GPUs, enabling more responsive and cost-efficient on-device LLM inference for end-user devices.

Abstract

Deploying large language models (LLMs) on end-user devices is gaining importance due to benefits in responsiveness, privacy, and operational cost. Yet the limited memory and compute capability of mobile and desktop GPUs make efficient execution difficult. Recent observations suggest that the internal activations of LLMs are often dynamically sparse, meaning that for each input, only part of the network contributes significantly to the output. Such sparsity could reduce computation, but it interacts poorly with group-wise quantization, which remains the dominant approach for fitting LLMs onto resource-constrained hardware. To reconcile these two properties, this study proposes a set of techniques that realize dynamic sparse inference under low-bit quantization. The method features: (1) a zigzag-patterned quantization layout that organizes weights in a way consistent with activation sparsity and improves GPU memory locality; (2) a specialized GEMV kernel designed for this layout to fully utilize parallel compute units; and (3) a compact runtime mechanism that gathers sparse indices with minimal overhead. Across several model scales and hardware configurations, the approach achieves up to 1.55x faster decoding throughput while maintaining accuracy comparable to dense quantized inference, showing that structured sparsity and quantization can effectively coexist on commodity GPUs.

Enabling Dynamic Sparsity in Quantized LLM Inference

TL;DR

This paper addresses the challenge of running quantized LLMs on client devices by leveraging dynamic activation sparsity without sacrificing the benefits of low-bit quantization. It introduces SpQt, a co-design consisting of a Zigzag weight layout aligned with activation sparsity and a sparsity-aware Zigzag GEMV kernel accompanied by a runtime sparse index collector. The key contributions are the Zigzag layout, the specialized kernels and index-collection mechanism, and an end-to-end system validated on Apple GPUs across Llama-2/3 models, achieving up to 1.55× end-to-end speedups and up to 1.8× GFLOPS improvements with minimal accuracy loss. This work demonstrates that structured sparsity and quantization can coexist effectively on commodity GPUs, enabling more responsive and cost-efficient on-device LLM inference for end-user devices.

Abstract

Deploying large language models (LLMs) on end-user devices is gaining importance due to benefits in responsiveness, privacy, and operational cost. Yet the limited memory and compute capability of mobile and desktop GPUs make efficient execution difficult. Recent observations suggest that the internal activations of LLMs are often dynamically sparse, meaning that for each input, only part of the network contributes significantly to the output. Such sparsity could reduce computation, but it interacts poorly with group-wise quantization, which remains the dominant approach for fitting LLMs onto resource-constrained hardware. To reconcile these two properties, this study proposes a set of techniques that realize dynamic sparse inference under low-bit quantization. The method features: (1) a zigzag-patterned quantization layout that organizes weights in a way consistent with activation sparsity and improves GPU memory locality; (2) a specialized GEMV kernel designed for this layout to fully utilize parallel compute units; and (3) a compact runtime mechanism that gathers sparse indices with minimal overhead. Across several model scales and hardware configurations, the approach achieves up to 1.55x faster decoding throughput while maintaining accuracy comparable to dense quantized inference, showing that structured sparsity and quantization can effectively coexist on commodity GPUs.

Paper Structure

This paper contains 44 sections, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background: Sparsity in LLM inference
  3. Problem & our techniques
  4. Results
  5. Contributions
  6. Motivations
  7. Dynamic Sparsity: Neuron vs. Activation
  8. Conflict between dynamic sparsity and quantization
  9. Overview
  10. System model
  11. The Operations
  12. Applicability
  13. Design
  14. New Zigzag quantization weight layout
  15. Primer on Quantization
  16. ...and 29 more sections

Figures (10)

  • Figure 1: Our zigzag quantization layout and GPU kernel co-design enables structured sparse computation. (a) Default row-wise layout conflicts with activation sparsity and causes irregular GPU execution. (b) Our Zigzag layout aligns with column-wise activaiton sparsity, enabling efficient structured processing on GPU.
  • Figure 2: Comparison between (a) neuron sparsity and (b) activation sparsity.
  • Figure 3: System architecture of SpQt and workflow.
  • Figure 4: Our weight layout enables structural skipping of the computation and works more efficient with the activation sparsity compares to default weight layout in GPU processing
  • Figure 5: Our kernel introduces new parallelism scheme on the column dimension on top of the Zigzag weight layout, creates more flexibility on parallelism.
  • ...and 5 more figures