Experimenting a New Programming Practice with LLMs
Simiao Zhang, Jiaping Wang, Guoliang Dong, Jun Sun, Yueling Zhang, Geguang Pu
TL;DR
This work tackles the challenge of deriving non-trivial software from vague requirements by introducing AISD, an AI-aided development framework that keeps humans involved in requirement analysis, system design, and validation. AISD uses a structured, iterative workflow—generating use cases, proposing a simple system design, and automating coding and testing with human feedback to refine the result. A novel CAASD benchmark is proposed to objectively assess AI-driven development capabilities, enabling comparisons with state-of-the-art baselines. Empirical results show AISD achieves a higher pass rate (around 75%) while consuming fewer tokens than baselines, underscoring the value of human-in-the-loop interaction for aligning AI outputs with user expectations and practical needs.
Abstract
The recent development on large language models makes automatically constructing small programs possible. It thus has the potential to free software engineers from low-level coding and allow us to focus on the perhaps more interesting parts of software development, such as requirement engineering and system testing. In this project, we develop a prototype named AISD (AI-aided Software Development), which is capable of taking high-level (potentially vague) user requirements as inputs, generates detailed use cases, prototype system designs, and subsequently system implementation. Different from existing attempts, AISD is designed to keep the user in the loop, i.e., by repeatedly taking user feedback on use cases, high-level system designs, and prototype implementations through system testing. AISD has been evaluated with a novel benchmark of non-trivial software projects. The experimental results suggest that it might be possible to imagine a future where software engineering is reduced to requirement engineering and system testing only.