Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A method for accelerating low precision operations by sparse matrix multiplication

Hongyaoxing Gu

TL;DR

Improvements in the methodology are proposed by incorporating low-precision quantization and employing a residual matrix for error correction and combines vector-wise quantization method which can effectively control the quantization error while maintaining high acceleration effect.

Abstract

In recent years, the fervent demand for computational power across various domains has prompted hardware manufacturers to introduce specialized computing hardware aimed at enhancing computational capabilities. Particularly, the utilization of tensor hardware supporting low precision has gained increasing prominence in scientific research. However, the use of low-precision tensor hardware for computational acceleration often introduces errors, posing a fundamental challenge of simultaneously achieving effective acceleration while maintaining computational accuracy. This paper proposes improvements in the methodology by incorporating low-precision quantization and employing a residual matrix for error correction and combines vector-wise quantization method.. The key innovation lies in the use of sparse matrices instead of dense matrices when compensating for errors with a residual matrix. By focusing solely on values that may significantly impact relative errors under a specified threshold, this approach aims to control quantization errors while reducing computational complexity. Experimental results demonstrate that this method can effectively control the quantization error while maintaining high acceleration effect.The improved algorithm on the CPU can achieve up to 15\% accuracy improvement while 1.46 times speed improvement.

A method for accelerating low precision operations by sparse matrix multiplication

TL;DR

Improvements in the methodology are proposed by incorporating low-precision quantization and employing a residual matrix for error correction and combines vector-wise quantization method which can effectively control the quantization error while maintaining high acceleration effect.

Abstract

In recent years, the fervent demand for computational power across various domains has prompted hardware manufacturers to introduce specialized computing hardware aimed at enhancing computational capabilities. Particularly, the utilization of tensor hardware supporting low precision has gained increasing prominence in scientific research. However, the use of low-precision tensor hardware for computational acceleration often introduces errors, posing a fundamental challenge of simultaneously achieving effective acceleration while maintaining computational accuracy. This paper proposes improvements in the methodology by incorporating low-precision quantization and employing a residual matrix for error correction and combines vector-wise quantization method.. The key innovation lies in the use of sparse matrices instead of dense matrices when compensating for errors with a residual matrix. By focusing solely on values that may significantly impact relative errors under a specified threshold, this approach aims to control quantization errors while reducing computational complexity. Experimental results demonstrate that this method can effectively control the quantization error while maintaining high acceleration effect.The improved algorithm on the CPU can achieve up to 15\% accuracy improvement while 1.46 times speed improvement.
Paper Structure (32 sections, 37 equations, 8 figures, 7 tables, 1 algorithm)

This paper contains 32 sections, 37 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Low precision computation and sparse matrix acceleration
  4. Low precision GEMM
  5. Sparse matrix acceleration
  6. Quantization and Dequantization
  7. Quantized GEMM and residual-based refinement method
  8. Quantization methods of the input matrix
  9. Error compensating in output matrix
  10. Sparse quant
  11. Sparse residual error compensation algorithm
  12. Methods to obtain sparsity
  13. Example
  14. Algorithm
  15. Vector-wise Quantization
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Figures(a) includes the efficiency of (int8-int32), and (int4-int32) executions. Here, 'cb' represents Cublas, 'ct' represents Cutlass, and 'tc' denotes the use of Tensor Cores. Figures(c) show the execution time of GEMM with different precision in different math libraries under Intel(R) Xeon(R) Platinum 8163. Where E stands for EIGEN mathematical library and M stands for MKL mathematical library. I8 stands for int8 as the calculation accuracy and int32 as the result accuracy(Eigen does not support the calculation). Figures(b) and Figures(d) shows the acceleration of sparse matrix multiplication (SPMM) over dense matrix multiplication (GEMM) on GPU/CPU. $\eta$ indicates the density of SPMM with the same speed as GEMM.
  • Figure 2: Representation of integer quantization
  • Figure 3: Normal precision acceleration (a), and low precision quantization acceleration processes(b)
  • Figure 4: Comparison of the effect of different quantization methods
  • Figure 5: Workflow of Algorithm
  • ...and 3 more figures