Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study

TL;DR

This replication study extends Guo et al.'s work by evaluating two open-source, small-scale LLMs—CodeLlama (7B) and Llama 2 (7B)—on automated code refinement tasks using the CR and CRN datasets. By exploring prompt strategies and temperatures, and comparing against ChatGPT and CodeReviewer, the authors show that with temperature set to 0 and carefully crafted prompts, CodeLlama can achieve BLEU-T scores and overall performance close to ChatGPT, especially for code-centric tasks like refactoring. The work highlights the practical viability of privacy-preserving, cost-effective open-source models for code refinement, while also acknowledging limitations with external-context reliance, dataset quality, and the persistent gap in exact-match capability. It further evaluates Llama 3.1, finding it does not outperform CodeLlama in code refinement, underscoring the continued advantage of task-tuned, open-source models for this domain. Overall, the study supports deploying open-source LLMs for code review assistance, given careful prompt engineering, local hosting, and an awareness of task-type strengths and weaknesses.

Abstract

Code reviews are an integral part of software development and have been recognized as a crucial practice for minimizing bugs and favouring higher code quality. They serve as an important checkpoint before committing code and play an essential role in knowledge transfer between developers. However, code reviews can be time-consuming and can stale the development of large software projects. In a recent study, Guo et al. assessed how ChatGPT3.5 can help the code review process. They evaluated the effectiveness of ChatGPT in automating the code refinement tasks, where developers recommend small changes in the submitted code. While Guo et al. 's study showed promising results, proprietary models like ChatGPT pose risks to data privacy and incur extra costs for software projects. In this study, we explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2 (7B parameters). Our results show that, if properly tuned, the Llama models, particularly CodeLlama, can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. However, not all code refinement tasks are equally successful: tasks that require changing existing code (e.g., refactoring) are more manageable for models to automate than tasks that demand new code. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.

Figures (3)

• Figure 1: Study Design Workflow. CR stands for Code Review dataset and CRN stands for Code Review-New dataset. Experiments for RQ1 are repeated twice on the CRN train/validation sets, and twice for RQ2 on both datasets.
• Figure 2: Sample from the CR dataset
• Figure 3: Prompt generation example for Llama models using prompt I and II templates.