Achieving Peak Performance for Large Language Models: A Systematic Review

TL;DR

This systematic literature review analyzes optimization and acceleration techniques for large language models ranging from billions to trillions of parameters. It categorizes approaches into training optimization, hardware optimization, and scalability/reliability, and surveys a broad set of libraries/frameworks (e.g., GPipe, Megatron-LM, DeepSpeed Inference, FlexGen, PETALS) with their trade-offs in performance, cost, and scalability. The study provides two in-depth case studies (SparseGPT pruning and QMoE compression) to illustrate practical resource-constraint solutions, and it concludes with a roadmap of future directions toward resource-efficient, modular, and scalable LLM systems. The findings highlight significant gains from distributed and heterogeneous training, advanced memory management, model compression, and hardware-aware offloading, underscoring practical pathways to deploy trillion-parameter models. Overall, the work serves as a consolidated guide for researchers and practitioners aiming to optimize LLM training, inference, and serving in real-world settings.

Abstract

In recent years, large language models (LLMs) have achieved remarkable success in natural language processing (NLP). LLMs require an extreme amount of parameters to attain high performance. As models grow into the trillion-parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models. Optimizing LLM performance involves two main approaches: fine-tuning pre-trained models for specific tasks to achieve state-of-the-art performance, and reducing costs or improving training time while maintaining similar performance. This paper presents a systematic literature review (SLR) following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We reviewed 65 publications out of 983 from 2017 to December 2023, retrieved from 5 databases. The study presents methods to optimize and accelerate LLMs while achieving cutting-edge results without sacrificing accuracy. We begin with an overview of the development of language modeling, followed by a detailed explanation of commonly used frameworks and libraries, and a taxonomy for improving and speeding up LLMs based on three classes: LLM training, LLM inference, and system serving. We then delve into recent optimization and acceleration strategies such as training optimization, hardware optimization, scalability and reliability, accompanied by the taxonomy and categorization of these strategies. Finally, we provide an in-depth comparison of each class and strategy, with two case studies on optimizing model training and enhancing inference efficiency. These case studies showcase practical approaches to address LLM resource limitations while maintaining performance.