Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Early Exit Deep Neural Networks in NLP

Divya Jyoti Bajpai, Manjesh Kumar Hanawal

TL;DR

Large DNNs in NLP pose latency and resource challenges, motivating Early Exit DNNs (EEDNNs) that attach intermediate classifiers to enable adaptive, anytime inference. The survey consolidates design choices for exits, including confidence metrics, thresholding, and training regimes (separate vs joint), and analyzes their impact across NLP tasks and Vision-Language models. It highlights key applications such as text classification, NLI, translation, summarization, and sequence labeling, as well as domain generalization and edge-cloud deployment, OOD detection, and reinforcement-learning contexts. The work provides practical guidance and identifies open challenges—exit criteria, robust confidence estimation, and domain adaptation—to advance efficient, robust NLP systems using early exits.

Abstract

Deep Neural Networks (DNNs) have grown increasingly large in size to achieve state of the art performance across a wide range of tasks. However, their high computational requirements make them less suitable for resource-constrained applications. Also, real-world datasets often consist of a mixture of easy and complex samples, necessitating adaptive inference mechanisms that account for sample difficulty. Early exit strategies offer a promising solution by enabling adaptive inference, where simpler samples are classified using the initial layers of the DNN, thereby accelerating the overall inference process. By attaching classifiers at different layers, early exit methods not only reduce inference latency but also improve the model robustness against adversarial attacks. This paper presents a comprehensive survey of early exit methods and their applications in NLP.

A Survey of Early Exit Deep Neural Networks in NLP

TL;DR

Large DNNs in NLP pose latency and resource challenges, motivating Early Exit DNNs (EEDNNs) that attach intermediate classifiers to enable adaptive, anytime inference. The survey consolidates design choices for exits, including confidence metrics, thresholding, and training regimes (separate vs joint), and analyzes their impact across NLP tasks and Vision-Language models. It highlights key applications such as text classification, NLI, translation, summarization, and sequence labeling, as well as domain generalization and edge-cloud deployment, OOD detection, and reinforcement-learning contexts. The work provides practical guidance and identifies open challenges—exit criteria, robust confidence estimation, and domain adaptation—to advance efficient, robust NLP systems using early exits.

Abstract

Deep Neural Networks (DNNs) have grown increasingly large in size to achieve state of the art performance across a wide range of tasks. However, their high computational requirements make them less suitable for resource-constrained applications. Also, real-world datasets often consist of a mixture of easy and complex samples, necessitating adaptive inference mechanisms that account for sample difficulty. Early exit strategies offer a promising solution by enabling adaptive inference, where simpler samples are classified using the initial layers of the DNN, thereby accelerating the overall inference process. By attaching classifiers at different layers, early exit methods not only reduce inference latency but also improve the model robustness against adversarial attacks. This paper presents a comprehensive survey of early exit methods and their applications in NLP.
Paper Structure (18 sections, 1 equation, 4 figures)

This paper contains 18 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Advantages of EEDNNs
  3. Areas of research
  4. Foundation of Early Exit DNNs
  5. Setup
  6. Training methods
  7. Defining confidence
  8. Choice of thresholds
  9. Inference
  10. Applications
  11. Text classification and NLI tasks
  12. Text Summarization
  13. Sequence labeling tasks
  14. Language Translation
  15. Vision-language tasks
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Difference between the DNN and EEDNN.
  • Figure 2: The figure shows the average of the confidence values over the true class across all the layers for the SST-2 dataset.
  • Figure 3: Separate training vs Joint Training
  • Figure 4: Inference methods: 1) Max Probability: confidence is the maximum output of an individual classifier. 2) Patience-based: relies on prediction consistency between classifiers. 3) Ensemble: aggregates weighted results from multiple classifiers.