Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, Houari Sahraoui

TL;DR

This paper tackles the high computational cost of fine-tuning large language models for code generation by evaluating parameter-efficient fine-tuning (PEFT) methods. It systematically compares PEFT with In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) across Conala, CodeAlpacaPy, and APPS using multiple model families, including CodeLlama variants. The results show that LoRA generally provides the strongest gains among PEFT methods, while QLoRA enables scaling to very large models within a 24 GB GPU budget and can outperform alternatives in several settings, including APPS. These findings suggest PEFT can democratize access to high-quality code generation with LLMs and broaden practical software-engineering applications, with code and configurations openly available at the provided repository.

Abstract

Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee parameter-efficient fine-tuning (PEFT) as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs in the context of automated code generation. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL and RAG across a diverse set of LLMs and three representative Python code generation datasets: Conala, CodeAlpacaPy, and APPS. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at https://github.com/martin-wey/peft-llm-code/.

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

TL;DR

This paper tackles the high computational cost of fine-tuning large language models for code generation by evaluating parameter-efficient fine-tuning (PEFT) methods. It systematically compares PEFT with In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) across Conala, CodeAlpacaPy, and APPS using multiple model families, including CodeLlama variants. The results show that LoRA generally provides the strongest gains among PEFT methods, while QLoRA enables scaling to very large models within a 24 GB GPU budget and can outperform alternatives in several settings, including APPS. These findings suggest PEFT can democratize access to high-quality code generation with LLMs and broaden practical software-engineering applications, with code and configurations openly available at the provided repository.

Abstract

Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee parameter-efficient fine-tuning (PEFT) as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs in the context of automated code generation. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL and RAG across a diverse set of LLMs and three representative Python code generation datasets: Conala, CodeAlpacaPy, and APPS. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at https://github.com/martin-wey/peft-llm-code/.
Paper Structure (20 sections, 2 equations, 7 figures, 4 tables)

This paper contains 20 sections, 2 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG)
  4. Applying LLMs with limited resources
  5. Methodology
  6. Research Questions
  7. Datasets and Task
  8. ICL and RAG
  9. Small and Large Language Models
  10. Metrics
  11. Implementation Details
  12. Experimental Results
  13. RQ1: Baseline Effectiveness of Models Using Zero-Shot and ICL
  14. RQ2: Effectiveness of Models using PEFT Techniques
  15. RQ3: Comparative Analysis of LoRA, ICL, and RAG
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Peak GPU memory consumption during models fine-tuning using full fine-tuning (ft), LoRA, and QLoRA.
  • Figure 2: Token length distribution of the Conala, CodeAlpacaPy, and APPS datasets.
  • Figure 3: [RQ1] -- Effectiveness of the models using ICL with various number of random examples on the Conala and CodeAlpacaPy datasets.
  • Figure 4: [RQ2] -- Effectiveness of the models fine-tuned using LoRA for both datasets in terms of EM@10 and CodeBLEU.
  • Figure 5: [RQ2] -- Effectiveness and GPU usage of 7B, 13B, and 34B CodeLlama-Python (CL) LLMs fine-tuned using LoRA and QLoRA with 8-bit and 4-bit quantization.
  • ...and 2 more figures