Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference

Daya Khudia, Jianyu Huang, Protonu Basu, Summer Deng, Haixin Liu, Jongsoo Park, Mikhail Smelyanskiy

TL;DR

The paper presents fbGEMM, a CPU-focused library for high-performance quantized inference that fuses quantization steps with a highly optimized, shape-aware GEMM implementation. By using affine quantization, prepacking weights, and modular, fusion-friendly building blocks, fbGEMM achieves substantial speedups (over 2x) on real-world Facebook workloads like translation and content understanding. It demonstrates end-to-end pipelines with INT8 GEMMs and INT16 accumulation, including outlier-aware quantization and dynamic kernel generation for diverse matrix shapes. The work suggests that tight integration of packing, quantization, and post-gemm processing can substantially improve efficiency and may inform future HPC GEMM interfaces.

Abstract

Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.

FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference

TL;DR

The paper presents fbGEMM, a CPU-focused library for high-performance quantized inference that fuses quantization steps with a highly optimized, shape-aware GEMM implementation. By using affine quantization, prepacking weights, and modular, fusion-friendly building blocks, fbGEMM achieves substantial speedups (over 2x) on real-world Facebook workloads like translation and content understanding. It demonstrates end-to-end pipelines with INT8 GEMMs and INT16 accumulation, including outlier-aware quantization and dynamic kernel generation for diverse matrix shapes. The work suggests that tight integration of packing, quantization, and post-gemm processing can substantially improve efficiency and may inform future HPC GEMM interfaces.

Abstract

Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.

Paper Structure

This paper contains 10 sections, 2 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Understanding low-precision GEMM
  4. FBGEMM's key features
  5. Showcasing a sample end-to-end pipeline
  6. Conclusion
  7. FBGEMM Interface
  8. Modular building blocks to construct a pipeline in FBGEMM
  9. Shape specific optimizations
  10. Performance Experiments

Figures (5)

  • Figure 1: A sample end-to-end pipeline to illustrate INT8 matrix multiplication with INT16 accumulation in fbgemm.
  • Figure 2: Modular building blocks to construct a pipeline in fbgemm.
  • Figure 3: Common matrix multiplication shapes in popular deep neural networks.
  • Figure 4: Performance for the compute-bound shapes with INT8 matrix multiplication and INT16 accumulation in fbgemm.
  • Figure 5: Performance for the bandwidth-bound shapes with INT8 matrix multiplication and INT32 accumulation in fbgemm.