Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Large Language Models for Software Engineering

Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, Zhenyu Chen

TL;DR

This paper surveys the rapidly growing intersection of large language models (LLMs) and software engineering (SE). It provides a taxonomy of 62 code-focused LLMs across encoder-only, encoder-decoder, and decoder-only architectures, catalogs 15 pre-training objectives, and maps 16 downstream SE tasks, drawing on 1009 studies up to Aug 2024. It analyzes how LLMs are evaluated, benchmarked, and integrated into SE workflows, and discusses domain tuning, compression, security, and reproducibility via open artifacts. The work highlights data leakage and evaluation challenges, advocates domain-specific LLMs and clean benchmarks, and offers a practical roadmap and living artifacts repository to guide future research and practical deployment.

Abstract

Software Engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing Large Language Models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information of the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 947 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing and distillation. Finally, we highlight several challenges and potential opportunities on applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository: https://github.com/iSEngLab/AwesomeLLM4SE.

A Survey on Large Language Models for Software Engineering

TL;DR

This paper surveys the rapidly growing intersection of large language models (LLMs) and software engineering (SE). It provides a taxonomy of 62 code-focused LLMs across encoder-only, encoder-decoder, and decoder-only architectures, catalogs 15 pre-training objectives, and maps 16 downstream SE tasks, drawing on 1009 studies up to Aug 2024. It analyzes how LLMs are evaluated, benchmarked, and integrated into SE workflows, and discusses domain tuning, compression, security, and reproducibility via open artifacts. The work highlights data leakage and evaluation challenges, advocates domain-specific LLMs and clean benchmarks, and offers a practical roadmap and living artifacts repository to guide future research and practical deployment.

Abstract

Software Engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing Large Language Models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information of the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 947 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing and distillation. Finally, we highlight several challenges and potential opportunities on applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository: https://github.com/iSEngLab/AwesomeLLM4SE.
Paper Structure (74 sections, 6 figures, 6 tables)

This paper contains 74 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Survey Methodology
  3. Research Questions
  4. Search Strategy
  5. Study Selection
  6. Trend Observation
  7. RQ1: LLM Perspective
  8. RQ1.1: What LLMs have been released to support SE?
  9. Encoder-only LLMs
  10. Encoder-decoder LLMs
  11. Decoder-only LLMs
  12. RQ1.2: How are LLMs used in pre-training tasks?
  13. Code Sequence Modeling and Prediction
  14. Bidirectional Understanding and Generation between Code and Natural Language
  15. Code Structure and Relationship Understanding
  16. ...and 59 more sections

Figures (6)

  • Figure 1: Number of collected papers over years
  • Figure 2: Taxonomy of RQ1
  • Figure 3: Taxonomy of the application of LLMs in different domains within software engineering
  • Figure 4: Taxonomy of the application of LLMs in different domains within software engineering (Continue)
  • Figure 5: Taxonomy of the application of LLMs in different domains within software engineering (Continue)
  • ...and 1 more figures