Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference

Wanli Zhong, Haibo Feng, Zirui Zhou, Hanyang Peng, Shiqi Yu

TL;DR

IntAttention tackles the bottleneck of quantized attention on edge devices by delivering a fully integer, retraining-free pipeline. Its core, IndexSoftmax, uses integer-domain clipping and a LUT-based exponential surrogate to replace floating-point softmax, enabling end-to-end integer dataflow from QK^T to PV. Across language and vision benchmarks on ARM edge hardware, it achieves up to 3.7x speedup and 61% energy reduction with accuracy comparable to baselines. This approach shifts the bottleneck away from softmax to GEMM kernels, offering practical, deployable Transformer inference on commodity devices.

Abstract

Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax as the dominant bottleneck. This stage incurs a costly dequantize-softmax-requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer, plug-and-play attention pipeline without retraining. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup-table approximation, and direct integer normalization, thereby eliminating all datatype conversion overhead. We evaluate IntAttention and demonstrate consistent and substantial gains. Our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and 2.0x faster than conventional INT8 attention pipelines on Armv8 CPUs. These gains are achieved with high-fidelity accuracy comparable to baselines across diverse language and vision models, enabling practical and efficient Transformer inference on commodity edge devices. Code will be released in later version of this work.

IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference

TL;DR

IntAttention tackles the bottleneck of quantized attention on edge devices by delivering a fully integer, retraining-free pipeline. Its core, IndexSoftmax, uses integer-domain clipping and a LUT-based exponential surrogate to replace floating-point softmax, enabling end-to-end integer dataflow from QK^T to PV. Across language and vision benchmarks on ARM edge hardware, it achieves up to 3.7x speedup and 61% energy reduction with accuracy comparable to baselines. This approach shifts the bottleneck away from softmax to GEMM kernels, offering practical, deployable Transformer inference on commodity devices.

Abstract

Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax as the dominant bottleneck. This stage incurs a costly dequantize-softmax-requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer, plug-and-play attention pipeline without retraining. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup-table approximation, and direct integer normalization, thereby eliminating all datatype conversion overhead. We evaluate IntAttention and demonstrate consistent and substantial gains. Our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and 2.0x faster than conventional INT8 attention pipelines on Armv8 CPUs. These gains are achieved with high-fidelity accuracy comparable to baselines across diverse language and vision models, enabling practical and efficient Transformer inference on commodity edge devices. Code will be released in later version of this work.

Paper Structure

This paper contains 42 sections, 18 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Attention and Dynamic Quantization
  4. Scaled Dot-Product Attention
  5. Dynamic Quantizing Q,K,V
  6. Integer accumulation and scaling.
  7. Emerging Bottleneck in Quantized Attention Pipelines
  8. Numerically stable softmax.
  9. Cost drivers on edge hardware.
  10. Measured breakdown and implication.
  11. Context in prior work.
  12. Acceleration Strategies
  13. Hardware-oriented softmax co-design.
  14. Input-aware quantization and LUT-based softmax.
  15. Integer-only softmax in Transformer quantization.
  16. ...and 27 more sections

Figures (9)

  • Figure 1: Comparison between conventional quantized attention and the proposed IntAttention, where IntAttention maintains an end-to-end integer dataflow from $\mathbf{QK}^\top$ to $\mathbf{PV}$.
  • Figure 2: Breakdown of time share for the $\mathrm{dequantize}\rightarrow\mathrm{softmax}\rightarrow\mathrm{requantize}$ path across different precisions. Once GEMMs are accelerated to INT8, this path emerges as the dominant latency and becomes the next optimization target.
  • Figure 3: Overview of the proposed IntAttention pipeline.
  • Figure 4: Illustration of the exponential activation in Softmax. Most logits lie in the near-zero region, where $e^{x}$ contributes negligibly to the normalization. Only a small subset of higher logits significantly affects the output distribution.
  • Figure 5: IndexSoftmax achieves 4x higher LUT resolution under the same memory budget, enabling higher-fidelity exponential approximation without dynamic clipping or global statistics, which are costly on edge devices.
  • ...and 4 more figures