Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastFormers: Highly Efficient Transformer Models for Natural Language Understanding

Young Jin Kim, Hany Hassan Awadalla

TL;DR

FastFormers targets inference-time efficiency for Transformer-based NLU by combining knowledge distillation, structured pruning, and low-precision quantization with runtime optimizations. The paper demonstrates substantial speed-ups and energy savings on CPU and GPU across the SuperGLUE benchmark, supported by ablations and shared-task submissions. Key contributions include practical recipes for distillation and pruning, GPU/CPU quantization strategies, and graph- and runtime-level optimizations, all demonstrated on realistic deployment scenarios. The work offers a practical, open-source pathway for deploying cost-effective, energy-efficient Transformer models at scale.

Abstract

Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. We provide effective recipes that can guide practitioners to choose the best settings for various NLU tasks and pretrained models. Applying the proposed recipes to the SuperGLUE benchmark, we achieve from 9.8x up to 233.9x speed-up compared to out-of-the-box models on CPU. On GPU, we also achieve up to 12.4x speed-up with the presented methods. We show that FastFormers can drastically reduce cost of serving 100 million requests from 4,223 USD to just 18 USD on an Azure F16s_v2 instance. This translates to a sustainable runtime by reducing energy consumption 6.9x - 125.8x according to the metrics used in the SustaiNLP 2020 shared task.

FastFormers: Highly Efficient Transformer Models for Natural Language Understanding

TL;DR

FastFormers targets inference-time efficiency for Transformer-based NLU by combining knowledge distillation, structured pruning, and low-precision quantization with runtime optimizations. The paper demonstrates substantial speed-ups and energy savings on CPU and GPU across the SuperGLUE benchmark, supported by ablations and shared-task submissions. Key contributions include practical recipes for distillation and pruning, GPU/CPU quantization strategies, and graph- and runtime-level optimizations, all demonstrated on realistic deployment scenarios. The work offers a practical, open-source pathway for deploying cost-effective, energy-efficient Transformer models at scale.

Abstract

Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. We provide effective recipes that can guide practitioners to choose the best settings for various NLU tasks and pretrained models. Applying the proposed recipes to the SuperGLUE benchmark, we achieve from 9.8x up to 233.9x speed-up compared to out-of-the-box models on CPU. On GPU, we also achieve up to 12.4x speed-up with the presented methods. We show that FastFormers can drastically reduce cost of serving 100 million requests from 4,223 USD to just 18 USD on an Azure F16s_v2 instance. This translates to a sustainable runtime by reducing energy consumption 6.9x - 125.8x according to the metrics used in the SustaiNLP 2020 shared task.

Paper Structure

This paper contains 17 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Knowledge Distillation
  3. Knowledge distillation methods:
  4. Knowledge distillation results:
  5. Structured Pruning
  6. Structured pruning methods:
  7. Structured pruning results:
  8. Low Precision Inference
  9. 8-bit quantized matrix multiplications on the CPU:
  10. 16-bit model conversion for the GPU:
  11. Runtime Optimization
  12. Multi-processing optimization:
  13. Computational graph optimizations:
  14. Results
  15. Combined results
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Knowledge distillation methods
  • Figure 2: Structured pruning - inference time versus accuracy. Each data point indicates a pruned model with the number of remaining heads and the number of remaining intermediate hidden states of feed-forward layers.
  • Figure 3: Accuracy versus queries per second with various optimizations on CPU.