Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization

TL;DR

This paper addresses the lack of code-generation–specific prompt guidelines for large language models by deriving 10 empirical, actionable prompt-improvement guidelines through an automated, test-driven refinement loop across Python benchmarks and four LLMs. It then validates these guidelines with a survey of 50 practitioners, revealing patterns in usage and perceived usefulness that vary by scenario. The study provides practical implications for developers, educators, and tool-makers, and lays groundwork for automated prompt-recommenders that identify missing elements in prompts. By integrating benchmark-driven experimentation with human validation, the work advances how prompt design can be systematically guided in LLM-assisted software development. The approach highlights the value of explicit I/O formats, pre-/post-conditions, algorithmic detail, and examples for reliable code generation, offering a pathway to more predictable LLM behavior in real-world coding tasks.

Abstract

Large Language Models (LLMs) are nowadays extensively used for various types of software engineering tasks, primarily code generation. Previous research has shown how suitable prompt engineering could help developers in improving their code generation prompts. However, so far, there do not exist specific guidelines driving developers towards writing suitable prompts for code generation. In this work, we derive and evaluate development-specific prompt optimization guidelines. First, we use an iterative, test-driven approach to automatically refine code generation prompts, and we analyze the outcome of this process to identify prompt improvement items that lead to test passes. We use such elements to elicit 10 guidelines for prompt improvement, related to better specifying I/O, pre-post conditions, providing examples, various types of details, or clarifying ambiguities. We conduct an assessment with 50 practitioners, who report their usage of the elicited prompt improvement patterns, as well as their perceived usefulness, which does not always correspond to the actual usage before knowing our guidelines. Our results lead to implications not only for practitioners and educators, but also for those aimed at creating better LLM-aided software development tools.

Figures (4)

• Figure 1: Methodology Overview
• Figure 2: Original (top) and optimized (bottom) prompt. Example from the mbpp dataset, task id: 99, LLM used: GPT-4o mini
• Figure 3: Prompt improvement patterns usage frequency
• Figure 4: Perceived usefulness of the pattern improvement guidelines.