Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latency Adjustable Transformer Encoder for Language Understanding

Sajjad Kachuee, Mohammad Sharifkhani

TL;DR

This work tackles inference latency in Transformer-based language models by introducing a latency-adjustable encoder that prunes word-vectors during fine-tuning using the Attention Context Contribution (ACC) metric. It introduces Sort and Eliminate layers and an elimination rate tied to a fit-based metric and a Speedup Coefficient, enabling offline-tuning to adjust latency after training with minimal accuracy loss. Experiments across BERT-base, GPT-2, Flan-T5, Gemma2, Mistral, and Llama3 show up to about a 3x TTFT/latency speedup with small drops in accuracy or perplexity across tasks, text generation, and instruction tuning. The approach preserves the input’s global context and provides an analytical speedup estimate, making it practical for deployment on resource-constrained devices and real-time applications.

Abstract

Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of an efficient architecture. This paper proposes an efficient Transformer architecture that adjusts the inference computational cost adaptively with a desired inference latency speedup. In fine-tuning phase, the proposed method detects less important hidden sequence elements (word-vectors) and eliminates them in each encoder layer using a proposed Attention Context Contribution (ACC) metric. After the fine-tuning phase, with the novel offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections without any further training. Extensive experiments reveal that most word-vectors in higher Transformer layers contribute less to subsequent layers, allowing their removal to improve inference latency. Experimental results on various language understanding, text generation, and instruction tuning tasks and benchmarks demonstrate the approach's effectiveness across diverse datasets, with minimal impact on the input's global context. The technique improves Time-to-First-Token (TTFT) of Llama3 by up to 2.9x, with minor performance drop. The suggested approach posits that in Large Language Models (LLMs), although the complete network is necessary for training, it can be truncated during the fine-tuning phase.

Latency Adjustable Transformer Encoder for Language Understanding

TL;DR

This work tackles inference latency in Transformer-based language models by introducing a latency-adjustable encoder that prunes word-vectors during fine-tuning using the Attention Context Contribution (ACC) metric. It introduces Sort and Eliminate layers and an elimination rate tied to a fit-based metric and a Speedup Coefficient, enabling offline-tuning to adjust latency after training with minimal accuracy loss. Experiments across BERT-base, GPT-2, Flan-T5, Gemma2, Mistral, and Llama3 show up to about a 3x TTFT/latency speedup with small drops in accuracy or perplexity across tasks, text generation, and instruction tuning. The approach preserves the input’s global context and provides an analytical speedup estimate, making it practical for deployment on resource-constrained devices and real-time applications.

Abstract

Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of an efficient architecture. This paper proposes an efficient Transformer architecture that adjusts the inference computational cost adaptively with a desired inference latency speedup. In fine-tuning phase, the proposed method detects less important hidden sequence elements (word-vectors) and eliminates them in each encoder layer using a proposed Attention Context Contribution (ACC) metric. After the fine-tuning phase, with the novel offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections without any further training. Extensive experiments reveal that most word-vectors in higher Transformer layers contribute less to subsequent layers, allowing their removal to improve inference latency. Experimental results on various language understanding, text generation, and instruction tuning tasks and benchmarks demonstrate the approach's effectiveness across diverse datasets, with minimal impact on the input's global context. The technique improves Time-to-First-Token (TTFT) of Llama3 by up to 2.9x, with minor performance drop. The suggested approach posits that in Large Language Models (LLMs), although the complete network is necessary for training, it can be truncated during the fine-tuning phase.
Paper Structure (23 sections, 8 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 23 sections, 8 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Weight Pruning
  4. Head/Channel Pruning
  5. Layer/Block Pruning
  6. Token/Word-Vector Pruning
  7. Non-Pruning Methods
  8. Proposed Method
  9. Background
  10. Transformer Architecture Latency and TTFT Analysis
  11. Attention Context Contribution (ACC) metric
  12. Proposed Architecture
  13. Extending Proposed Architecture to Autoregressive Models
  14. Experiments
  15. Datasets
  16. ...and 8 more sections

Figures (7)

  • Figure 1: BERT-base devlin2018bert architecture with output layers
  • Figure 2: Obtaining Score Vector (SV) from Attention Probability Matrix in a given BERT devlin2018bert encoder layer
  • Figure 3: BERT-basedevlin2018bert Score Vector (SV) results on the IMDB sentiment analysis taskmaas2011learning
  • Figure 4: BERT-base devlin2018bert Encoder Attention Context Contribution (ACC) metric results (blue bars) and second degree fitted curves (red lines) on SST-2 socher2013recursive, MRPC dolan2005automatically, QQP iyer2017first, QNLI rajpurkar2016squad, RTE bentivogli2009fifth, MNLI williams2017broad, STS-B cer2017semeval and IMDB maas2011learning tasks.
  • Figure 5: The proposed self-attention layer
  • ...and 2 more figures