Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design
Giovanni Rosa, David Moreno-Lumbreras, Gregorio Robles, Jesús M. González-Barahona
TL;DR
The paper investigates how human guidance in specification and test-definition affects LLM-assisted code generation within a spec-driven, test-first workflow implemented by the Currante IDE extension. Using a Stage 1 Registered Report, it outlines a controlled study with LiveCodeBench problems, collecting fine-grained interaction logs and metrics to assess code correctness and the expressiveness of test-suite specifications. It contributes a reproducible experimental protocol, a telemetry schema for in-IDE evaluation, and open-source materials to guide future research and IDE design for human–AI collaboration in software engineering. The work aims to inform the design of next-generation development environments that better align human reasoning with model-driven code generation, enhancing reliability and developer trust in AI-assisted coding.
Abstract
Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their behavior in structured, specification-driven processes remains poorly understood. This paper presents an empirical study design using CURRANTE, a Visual Studio Code extension that enables a human-in-the-loop workflow for LLM-assisted code generation. The tool guides developers through three sequential stages--Specification, Tests, and Function--allowing them to define requirements, generate and refine test suites, and produce functions that satisfy those tests. Participants will solve medium-difficulty problems from the LiveCodeBench dataset, while the tool records fine-grained interaction logs, effectiveness metrics (e.g., pass rate, all-pass completion), efficiency indicators (e.g., time-to-pass), and iteration behaviors. The study aims to analyze how human intervention in specification and test refinement influences the quality and dynamics of LLM-generated code. The results will provide empirical insights into the design of next-generation development environments that align human reasoning with model-driven code generation.