Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Current Challenges of Software Engineering in the Era of Large Language Models

Cuiyun Gao, Xing Hu, Shan Gao, Xin Xia, Zhi Jin

TL;DR

This paper investigates the current challenges of integrating large language models into software engineering (LLM4SE) by conducting a qualitative study with 24 experts. It enumerates 26 challenges across seven SE aspects, spanning requirements/design, coding, testing, review, maintenance, vulnerability management, and data/training/evaluation, grounded in a structured SDLC lens. The authors detail the LLM4SE workflow—data construction, fine-tuning, prompting, and SE-specific LLMs—and discuss SE-specific tuning methods and prompting strategies. The work provides a practical roadmap and actionable insights to guide future research and practice toward trustworthy, reliable LLM-assisted software development.

Abstract

With the advent of large language models (LLMs) in the artificial intelligence (AI) area, the field of software engineering (SE) has also witnessed a paradigm shift. These models, by leveraging the power of deep learning and massive amounts of data, have demonstrated an unprecedented capacity to understand, generate, and operate programming languages. They can assist developers in completing a broad spectrum of software development activities, encompassing software design, automated programming, and maintenance, which potentially reduces huge human efforts. Integrating LLMs within the SE landscape (LLM4SE) has become a burgeoning trend, necessitating exploring this emergent landscape's challenges and opportunities. The paper aims at revisiting the software development life cycle (SDLC) under LLMs, and highlighting challenges and opportunities of the new paradigm. The paper first summarizes the overall process of LLM4SE, and then elaborates on the current challenges based on a through discussion. The discussion was held among more than 20 participants from academia and industry, specializing in fields such as software engineering and artificial intelligence. Specifically, we achieve 26 key challenges from seven aspects, including software requirement & design, coding assistance, testing code generation, code review, code maintenance, software vulnerability management, and data, training, and evaluation. We hope the achieved challenges would benefit future research in the LLM4SE field.

The Current Challenges of Software Engineering in the Era of Large Language Models

TL;DR

This paper investigates the current challenges of integrating large language models into software engineering (LLM4SE) by conducting a qualitative study with 24 experts. It enumerates 26 challenges across seven SE aspects, spanning requirements/design, coding, testing, review, maintenance, vulnerability management, and data/training/evaluation, grounded in a structured SDLC lens. The authors detail the LLM4SE workflow—data construction, fine-tuning, prompting, and SE-specific LLMs—and discuss SE-specific tuning methods and prompting strategies. The work provides a practical roadmap and actionable insights to guide future research and practice toward trustworthy, reliable LLM-assisted software development.

Abstract

With the advent of large language models (LLMs) in the artificial intelligence (AI) area, the field of software engineering (SE) has also witnessed a paradigm shift. These models, by leveraging the power of deep learning and massive amounts of data, have demonstrated an unprecedented capacity to understand, generate, and operate programming languages. They can assist developers in completing a broad spectrum of software development activities, encompassing software design, automated programming, and maintenance, which potentially reduces huge human efforts. Integrating LLMs within the SE landscape (LLM4SE) has become a burgeoning trend, necessitating exploring this emergent landscape's challenges and opportunities. The paper aims at revisiting the software development life cycle (SDLC) under LLMs, and highlighting challenges and opportunities of the new paradigm. The paper first summarizes the overall process of LLM4SE, and then elaborates on the current challenges based on a through discussion. The discussion was held among more than 20 participants from academia and industry, specializing in fields such as software engineering and artificial intelligence. Specifically, we achieve 26 key challenges from seven aspects, including software requirement & design, coding assistance, testing code generation, code review, code maintenance, software vulnerability management, and data, training, and evaluation. We hope the achieved challenges would benefit future research in the LLM4SE field.

Paper Structure

This paper contains 33 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Overall Process of LLM4SE
  4. Data Construction
  5. Data Collection
  6. Data Filtering
  7. Data Organization
  8. Fine-Tuning Techniques
  9. Adapter-based tuning.
  10. Low-rank adaptation.
  11. Prefix tuning.
  12. Prompt tuning.
  13. SE-Specific LLMs
  14. Prompting Techniques
  15. Downstream Tasks
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Overall process of LLM4SE.