Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Design

Yonghao Tan, Pingcheng Dong, Yongkun Wu, Yu Liu, Xuejiao Liu, Peng Luo, Shih-Yang Liu, Xijie Huang, Dong Zhang, Luhong Liang, Kwang-Ting Cheng

TL;DR

APSQ addresses the energy overhead caused by high-precision PSUMs in IS/WS dataflows by introducing Additive Partial Sum Quantization, which integrates PSUM accumulation into the quantization process. A grouping strategy and a Reconfigurable APSQ Engine (RAE) enable flexible, low-bit PSUM handling across various group sizes. Across NLP, CV, and LLM workloads, APSQ with INT8 PSUM storage achieves substantial energy savings (up to 31.7× in LLM decoding) with minimal accuracy loss (often below 1%). The work combines a PSUM-precision-aware analytical framework with hardware prototypes to demonstrate practical benefits for transformer-based accelerators.

Abstract

DNN accelerators, significantly advanced by model compression and specialized dataflow techniques, have marked considerable progress. However, the frequent access of high-precision partial sums (PSUMs) leads to excessive memory demands in architectures utilizing input/weight stationary dataflows. Traditional compression strategies have typically overlooked PSUM quantization, which may account for 69% of power consumption. This study introduces a novel Additive Partial Sum Quantization (APSQ) method, seamlessly integrating PSUM accumulation into the quantization framework. A grouping strategy that combines APSQ with PSUM quantization enhanced by a reconfigurable architecture is further proposed. The APSQ performs nearly lossless on NLP and CV tasks across BERT, Segformer, and EfficientViT models while compressing PSUMs to INT8. This leads to a notable reduction in energy costs by 28-87%. Extended experiments on LLaMA2-7B demonstrate the potential of APSQ for large language models. Code is available at https://github.com/Yonghao-Tan/APSQ.

APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Design

TL;DR

APSQ addresses the energy overhead caused by high-precision PSUMs in IS/WS dataflows by introducing Additive Partial Sum Quantization, which integrates PSUM accumulation into the quantization process. A grouping strategy and a Reconfigurable APSQ Engine (RAE) enable flexible, low-bit PSUM handling across various group sizes. Across NLP, CV, and LLM workloads, APSQ with INT8 PSUM storage achieves substantial energy savings (up to 31.7× in LLM decoding) with minimal accuracy loss (often below 1%). The work combines a PSUM-precision-aware analytical framework with hardware prototypes to demonstrate practical benefits for transformer-based accelerators.

Abstract

DNN accelerators, significantly advanced by model compression and specialized dataflow techniques, have marked considerable progress. However, the frequent access of high-precision partial sums (PSUMs) leads to excessive memory demands in architectures utilizing input/weight stationary dataflows. Traditional compression strategies have typically overlooked PSUM quantization, which may account for 69% of power consumption. This study introduces a novel Additive Partial Sum Quantization (APSQ) method, seamlessly integrating PSUM accumulation into the quantization framework. A grouping strategy that combines APSQ with PSUM quantization enhanced by a reconfigurable architecture is further proposed. The APSQ performs nearly lossless on NLP and CV tasks across BERT, Segformer, and EfficientViT models while compressing PSUMs to INT8. This leads to a notable reduction in energy costs by 28-87%. Extended experiments on LLaMA2-7B demonstrate the potential of APSQ for large language models. Code is available at https://github.com/Yonghao-Tan/APSQ.
Paper Structure (16 sections, 10 equations, 6 figures, 4 tables, 1 algorithm)

This paper contains 16 sections, 10 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Energy Efficiency Analysis
  4. Input Stationary
  5. Weight Stationary
  6. Quantization
  7. Method
  8. Additive Partial Sum Quantization
  9. Grouping Strategy
  10. Reconfigurable APSQ Architecture
  11. Experimental Results
  12. Experiment Setup
  13. Accuracy Analysis
  14. Hardware Performance Analysis
  15. Experiment on Large Language Model
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Energy breakdown of IS, WS, and OS dataflows for the BERT-Base model with 128 input token-length.
  • Figure 2: The analytical DNN accelerator and the Reconfigurable APSQ Engine (RAE) in this work.
  • Figure 3: (a) An example of pointwise convolution with tile-based computation scheme, (b) typical partial sum accumulation, (c) proposed INT8 additive partial sum quantization ($gs=1$).
  • Figure 4: Workflow of additive partial sum quantization with $gs=3$.
  • Figure 5: Normalized energy and accuracy across varied $gs$ settings for MRPC under WS dataflow on BERT-Base.
  • ...and 1 more figures