Towards Advancing Code Generation with Large Language Models: A Research Roadmap

TL;DR

The paper addresses the reliability, security, and practical evaluation gaps in LLM-based code generation. It introduces a six-layer architecture to decompose the generation process and a four-phase vision workflow that emphasizes human-in-the-loop, dynamic task management, and collaborative multi-agent setups. Key contributions include a structured framework, insights into technical and evaluation challenges, and actionable recommendations to improve robustness, usability, and security. The work aims to bridge research and real-world development, enabling more pragmatic, trustworthy LLM-assisted coding solutions.

Abstract

Recently, we have witnessed the rapid development of large language models, which have demonstrated excellent capabilities in the downstream task of code generation. However, despite their potential, LLM-based code generation still faces numerous technical and evaluation challenges, particularly when embedded in real-world development. In this paper, we present our vision for current research directions, and provide an in-depth analysis of existing studies on this task. We propose a six-layer vision framework that categorizes code generation process into distinct phases, namely Input Phase, Orchestration Phase, Development Phase, and Validation Phase. Additionally, we outline our vision workflow, which reflects on the currently prevalent frameworks. We systematically analyse the challenges faced by large language models, including those LLM-based agent frameworks, in code generation tasks. With these, we offer various perspectives and actionable recommendations in this area. Our aim is to provide guidelines for improving the reliability, robustness and usability of LLM-based code generation systems. Ultimately, this work seeks to address persistent challenges and to provide practical suggestions for a more pragmatic LLM-based solution for future code generation endeavors.

Figures (2)

• Figure 1: Proposed Six-Layer Architecture for LLM-Based Code Generation.
• Figure 2: Our vision workflow for LLM-Based Code Generation.