Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Transformers to LLMs: A Systematic Survey of Efficiency Considerations in NLP

Wazib Ansar, Saptarsi Goswami, Amlan Chakrabarti

TL;DR

The paper tackles the critical issue of efficiency in Transformer-based NLP and LLMs by conducting a comprehensive systematic literature review of 312 articles (2011–2025). It articulates a dual framework of model optimization (data curation, design, downsizing, dynamic inferencing) and LLM adaptation (pre-training, fine-tuning, prompting, RAG), paired with a robust evaluation using 13 benchmarks and multiple efficiency metrics. The study uncovers trends in scaling laws, energy and carbon footprints, and a shift toward open-weight models that promise lower costs and broader accessibility, while highlighting the tradeoffs between performance and efficiency. Overall, the work provides a detailed roadmap for achieving Pareto-efficient NLP systems and emphasizes the importance of sustainable practices across the model lifecycle.

Abstract

The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a systematic literature review on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles published between the years 2011 and 2025, efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis of the articles has been performed followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been conducted on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.

From Transformers to LLMs: A Systematic Survey of Efficiency Considerations in NLP

TL;DR

The paper tackles the critical issue of efficiency in Transformer-based NLP and LLMs by conducting a comprehensive systematic literature review of 312 articles (2011–2025). It articulates a dual framework of model optimization (data curation, design, downsizing, dynamic inferencing) and LLM adaptation (pre-training, fine-tuning, prompting, RAG), paired with a robust evaluation using 13 benchmarks and multiple efficiency metrics. The study uncovers trends in scaling laws, energy and carbon footprints, and a shift toward open-weight models that promise lower costs and broader accessibility, while highlighting the tradeoffs between performance and efficiency. Overall, the work provides a detailed roadmap for achieving Pareto-efficient NLP systems and emphasizes the importance of sustainable practices across the model lifecycle.

Abstract

The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a systematic literature review on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles published between the years 2011 and 2025, efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis of the articles has been performed followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been conducted on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.
Paper Structure (45 sections, 47 equations, 20 figures, 5 tables)

This paper contains 45 sections, 47 equations, 20 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Survey Methodology
  3. Selection of Survey Methodology
  4. Research Questions
  5. Selection Criteria
  6. Foundational Concepts
  7. Overview of NLP
  8. RQ1: Transformers in NLP
  9. Encoder-Decoder Architectures
  10. Attention Mechanism
  11. Transformer Architecture
  12. RQ2: Scaling of LLMs
  13. Transformer Parameters and Compute Calculation
  14. Laws for Scaling
  15. Estimating Energy Usage and Carbon Footprint
  16. ...and 30 more sections

Figures (20)

  • Figure 1: PRISMA of the survey
  • Figure 2: Applications of NLP
  • Figure 3: Illustration of the Multi-Head Attention
  • Figure 4: Illustration of the Transformer architecture (Inspired from Vaswani et al. vaswani2017attention)
  • Figure 5: Taxonomy of model optimization strategies
  • ...and 15 more figures