Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation

TL;DR

This paper investigates how to leverage large language models for code review automation by comparing fine-tuning against prompting strategies. Using GPT-3.5 and Magicoder, it evaluates performance across three code-review datasets with varying granularity, focusing on Exact Match and CodeBLEU metrics. Key findings show that fine-tuning GPT-3.5 produces the strongest improvements (EM gains up to 73-74% over zero-shot baselines), while in low-data scenarios few-shot prompting without a persona yields the best results among non-fine-tuned models. The study provides practical guidance on model selection, prompting design, and cost considerations for practitioners implementing LLM-based code review automation.

Abstract

Context: The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation. Objective: We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model's generation process without requiring a specific code review dataset. Method: We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT- 3.5 and Magicoder), and compare them with the Guo et al.'s approach and three existing code review automation approaches. Results: The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17% -74.23% higher EM than the Guo et al.'s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38% - 659.09% higher EM than GPT-3.5 with zero-shot learning. Conclusions: Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation.

Figures (8)

• Figure 1: An overview of the modelling pipeline of LLMs for code review automation.
• Figure 2: An overview of our experimental design (A persona is a part of zero-shot and few-shot learning).
• Figure 3: Prompt templates for zero-shot learning and few-shot learning that contain simple instructions (lang refers to a programming language). The text in blue is omitted when reviewers' comments are not used in experiments.
• Figure 4: (RQ3) Examples of the difference between code submitted for review and revised code generated by GPT-3.5 with zero-shot learning and few-shot learning.
• Figure 5: Example of the code changes of each type