Towards an Understanding of Large Language Models in Software Engineering Tasks

TL;DR

This paper surveys the use of large language models in software engineering tasks, assembling 123 post-2022 studies from six databases to map how LLMs are integrated into code-related activities. It classifies work into seven SE tasks and analyzes performance reports, revealing that LLMs reliably handle syntax-oriented tasks like code summarization and repair but struggle with semantics-driven tasks such as code generation and vulnerability detection. The authors provide a framework for organizing the field, compare models and evaluation approaches, and discuss limitations, benchmarks, and future directions. The study aims to guide researchers and practitioners in selecting or designing LLM-based tools for SE and highlights areas where improvements are most needed.

Abstract

Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in text generation and reasoning tasks. Derivative products, like ChatGPT, have been extensively deployed and highly sought after. Meanwhile, the evaluation and optimization of LLMs in software engineering tasks, such as code generation, have become a research focus. However, there is still a lack of systematic research on applying and evaluating LLMs in software engineering. Therefore, this paper comprehensively investigate and collate the research and products combining LLMs with software engineering, aiming to answer two questions: (1) What are the current integrations of LLMs with software engineering? (2) Can LLMs effectively handle software engineering tasks? To find the answers, we have collected related literature as extensively as possible from seven mainstream databases and selected 123 timely papers published starting from 2022 for analysis. We have categorized these papers in detail and reviewed the current research status of LLMs from the perspective of seven major software engineering tasks, hoping this will help researchers better grasp the research trends and address the issues when applying LLMs. Meanwhile, we have also organized and presented papers with evaluation content to reveal the performance and effectiveness of LLMs in various software engineering tasks, guiding researchers and developers to optimize.

Figures (2)

• Figure 1: Overview of methodology design.
• Figure 2: Number of literature on different software engineering tasks.