A Survey of Large Language Models

TL;DR

This survey provides a structured, comprehensive roadmap to large language models, tracing their evolution from statistical and neural language models to modern LLMs, with emphasis on scaling, emergent abilities, and practical deployment. It organizes current knowledge around four core areas—pre-training, adaptation tuning, utilization, and capacity evaluation—and couples this with a thorough review of data resources, architectures, and tooling. The authors consolidate resources, protocols, and empirical insights, offering guidance on data quality, curriculum design, efficient training, and alignment, while highlighting ongoing challenges in safety, hallucination, and knowledge recency. The work also surveys applications across NLP, information retrieval, multimodal systems, and domain-specific domains, and points to future directions such as sustained alignment, efficient adaptation, and robust evaluation. By providing a public GitHub repository of resources and experiments, the paper aims to accelerate reproducibility and practical progress in the rapidly evolving field of LLMs.

Abstract

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

Figures (19)

• Figure 1: The trends of the cumulative numbers of arXiv papers that contain the keyphrases "language model" (since June 2018) and "large language model" (since October 2019), respectively. The statistics are calculated using exact match by querying the keyphrases in title or abstract by months. We set different x-axis ranges for the two keyphrases, because "language models" have been explored at an earlier time. We label the points corresponding to important landmarks in the research progress of LLMs. A sharp increase occurs after the release of ChatGPT: the average number of published arXiv papers that contain "large language model" in title or abstract goes from 0.40 per day to 8.58 per day (Figure \ref{['fig:paper_number']}(b)).
• Figure 2: An evolution process of the four generations of language models (LM) from the perspective of task solving capacity. Note that the time period for each stage may not be very accurate, and we set the time mainly according to the publish date of the most representative studies at each stage. For neural language models, we abbreviate the paper titles of two representative studies to name the two approaches: NPLM Bengio-JMLR-2003-A ("A neural probabilistic language model") and NLPS Collobert-JMLR-2011 ("Natural language processing (almost) from scratch"). Due to the space limitation, we don't list all representative studies in this figure.
• Figure 3: A timeline of existing large language models (having a size larger than 10B) in recent years. The timeline was established mainly according to the release date (e.g., the submission date to arXiv) of the technical paper for a model. If there was no corresponding paper, we set the date of a model as the earliest time of its public release or announcement. We mark the LLMs with publicly available model checkpoints in yellow color. Due to the space limit of the figure, we only include the LLMs with publicly reported evaluation results.
• Figure 4: A brief illustration for the technical evolution of GPT-series models. We plot this figure mainly based on the papers, blog articles and official APIs from OpenAI. Here, solid lines denote that there exists an explicit evidence (e.g., the official statement that a new model is developed based on a base model) on the evolution path between two models, while dashed lines denote a relatively weaker evolution relation.
• Figure 5: An evolutionary graph of the research work conducted on LLaMA. Due to the huge number, we cannot include all the LLaMA variants in this figure, even much excellent work. To support incremental update, we share the source file of this figure, and welcome the readers to include the desired models by submitting the pull requests on our GitHub page.