Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code

TL;DR

The paper empirically compares GPT-4 prompting (basic, in-context, and task-specific) against 17 fine-tuned code-model baselines across code summarization, generation, and translation. It combines quantitative evaluation on CodeXGLUE, HumanEval, and MBPP with a qualitative user study involving academics and industry professionals to assess conversational prompting. Findings indicate that automated GPT-4 prompts do not universally outperform fine-tuned baselines, though task-specific prompts improve certain tasks, and conversational prompting with human feedback yields substantial performance gains. The work highlights the value of human-in-the-loop prompting for code tasks and discusses trade-offs between prompt engineering and fine-tuning, offering practical guidance and replication artifacts for future research.

Abstract

The rapid advancements in large language models (LLMs) have greatly expanded the potential for automated code-related tasks. Two primary methodologies are used in this domain: prompt engineering and fine-tuning. Prompt engineering involves applying different strategies to query LLMs, like ChatGPT, while fine-tuning further adapts pre-trained models, such as CodeBERT, by training them on task-specific data. Despite the growth in the area, there remains a lack of comprehensive comparative analysis between the approaches for code models. In this paper, we evaluate GPT-4 using three prompt engineering strategies -- basic prompting, in-context learning, and task-specific prompting -- and compare it against 17 fine-tuned models across three code-related tasks: code summarization, generation, and translation. Our results indicate that GPT-4 with prompt engineering does not consistently outperform fine-tuned models. For instance, in code generation, GPT-4 is outperformed by fine-tuned models by 28.3% points on the MBPP dataset. It also shows mixed results for code translation tasks. Additionally, a user study was conducted involving 27 graduate students and 10 industry practitioners. The study revealed that GPT-4 with conversational prompts, incorporating human feedback during interaction, significantly improved performance compared to automated prompting. Participants often provided explicit instructions or added context during these interactions. These findings suggest that GPT-4 with conversational prompting holds significant promise for automated code-related tasks, whereas fully automated prompt engineering without human involvement still requires further investigation.

Figures (3)

• Figure 1: Overall workflow of our study.
• Figure 2: Demographics from the survey.
• Figure 3: RQ2: Qualitative results.