Generative Software Engineering

TL;DR

The paper surveys how pre-trained models and large language models are applied to generative software engineering tasks, categorizing seven subtasks (requirements generation, code generation, code summarization, test generation, patch generation, code optimization, and code translation). It compiles and analyzes datasets, metrics, and methods across these subtasks, and identifies strengths, gaps, and promising directions for future research, including data augmentation, prompt design, domain knowledge integration, and robustness. The work consolidates a broad spectrum of benchmarks and approaches, from pattern-based RE techniques to advanced code-generation prompts and APR/repair methods, highlighting practical implications for accelerating SE tasks with LLMs. Overall, the review provides a structured roadmap for researchers and practitioners to leverage pre-trained models and LLMs in generative SE tasks, with emphasis on evaluation rigor and cross-task insights.

Abstract

The rapid development of deep learning techniques, improved computational power, and the availability of vast training data have led to significant advancements in pre-trained models and large language models (LLMs). Pre-trained models based on architectures such as BERT and Transformer, as well as LLMs like ChatGPT, have demonstrated remarkable language capabilities and found applications in Software engineering. Software engineering tasks can be divided into many categories, among which generative tasks are the most concern by researchers, where pre-trained models and LLMs possess powerful language representation and contextual awareness capabilities, enabling them to leverage diverse training data and adapt to generative tasks through fine-tuning, transfer learning, and prompt engineering. These advantages make them effective tools in generative tasks and have demonstrated excellent performance. In this paper, we present a comprehensive literature review of generative tasks in SE using pre-trained models and LLMs. We accurately categorize SE generative tasks based on software engineering methodologies and summarize the advanced pre-trained models and LLMs involved, as well as the datasets and evaluation metrics used. Additionally, we identify key strengths, weaknesses, and gaps in existing approaches, and propose potential research directions. This review aims to provide researchers and practitioners with an in-depth analysis and guidance on the application of pre-trained models and LLMs in generative tasks within SE.

Figures (4)

• Figure 1: Examples of requirement simulator pattern
• Figure 2: An example of Chatcoder
• Figure 3: An example of work process for extracting domain model
• Figure 4: An example of QuixBugs benchmark