Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework
Nathalia Nascimento, Everton Guimaraes, Paulo Alencar
TL;DR
The paper tackles the lack of standardized empirical evaluation for LLM-based code generation by proposing a bottom-up reference framework that structures experiments around problem sources, quality attributes, metrics, and related design factors. It grounds the framework in prior experience and a literature synthesis, and demonstrates its utility through mappings of representative studies, highlighting how non-determinism and prompting strategies can be incorporated as explicit design considerations. Key contributions include a modular six-component architecture, a clear taxonomy of problem sources and evaluation metrics, and a roadmap for future tooling, datasets, and cross-context applicability. The framework aims to enable more reproducible, comparable, and comprehensive experimentation across software engineering tasks that leverage LLMs.
Abstract
The rise of large language models (LLMs) has introduced transformative potential in automated code generation, addressing a wide range of software engineering challenges. However, empirical evaluation of LLM-based code generation lacks standardization, with studies varying widely in goals, tasks, and metrics, which limits comparability and reproducibility. In this paper, we propose a theoretical framework for designing and reporting empirical studies on LLM-based code generation. The framework is grounded in both our prior experience conducting such experiments and a comparative analysis of key similarities and differences among recent studies. It organizes evaluation around core components such as problem sources, quality attributes, and metrics, supporting structured and systematic experimentation. We demonstrate its applicability through representative case mappings and identify opportunities for refinement. Looking forward, we plan to evolve the framework into a more robust and mature tool for standardizing LLM evaluation across software engineering contexts.