Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study

André Storhaug, Jingyue Li

TL;DR

The paper tackles the high cost of fine-tuning large language models for unit test generation by systematically comparing full fine-tuning with several parameter-efficient fine-tuning (PEFT) methods (LoRA, (IA)3, and prompt tuning) across diverse code-focused models and real-world Java datasets. It shows that LoRA often matches full fine-tuning in unit-test quality, while prompt tuning offers the best cost efficiency for large models; however, tuned models can produce more non-executable tests, though executable tests tend to have better coverage and mutation-detecting capabilities. The study provides practical guidance: start with prompt tuning for cost savings, switch to LoRA for performance gains, and be mindful of compilation issues when generating executable tests. Overall, PEFT enables scalable, resource-efficient customization of LLMs for unit test generation with competitive outcomes and limited risk of catastrophic forgetting. The work advances both academia and industry by clarifying when and how to apply PEFT to improve automated software testing workflows.

Abstract

Parameter-efficient fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters, offer a promising solution by reducing the computational costs of tuning large language models (LLMs) while maintaining their performance. Existing studies have explored using PEFT and LLMs for various code-related tasks and found that the effectiveness of PEFT techniques is task-dependent. The state-of-the-art is limited to using LLMs with full fine-tuning to generate unit tests. The application of PEFT techniques in unit test generation remains underexplored. This paper investigates both full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt tuning, across thirteen models of different architectures and sizes. We use well-established benchmark datasets to evaluate their effectiveness in unit test generation and measure syntax correctness, CodeBLEU, pass@1, instruction coverage, branch coverage, and mutation score of the generated tests. Our findings show that LoRA can deliver performance comparable to full fine-tuning for unit test generation in several cases. If training costs are valued, prompt tuning is the most cost-effective approach, particularly for large models. However, the models tuned with full fine-tuning or PEFT may generate fewer executable test cases than the baseline model because they generate more tests calling nonexistent methods or having type mismatches. For the generated ones that are executable, the ones from the tuned models show better test coverage than those from the baseline model.

Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study

TL;DR

The paper tackles the high cost of fine-tuning large language models for unit test generation by systematically comparing full fine-tuning with several parameter-efficient fine-tuning (PEFT) methods (LoRA, (IA)3, and prompt tuning) across diverse code-focused models and real-world Java datasets. It shows that LoRA often matches full fine-tuning in unit-test quality, while prompt tuning offers the best cost efficiency for large models; however, tuned models can produce more non-executable tests, though executable tests tend to have better coverage and mutation-detecting capabilities. The study provides practical guidance: start with prompt tuning for cost savings, switch to LoRA for performance gains, and be mindful of compilation issues when generating executable tests. Overall, PEFT enables scalable, resource-efficient customization of LLMs for unit test generation with competitive outcomes and limited risk of catastrophic forgetting. The work advances both academia and industry by clarifying when and how to apply PEFT to improve automated software testing workflows.

Abstract

Parameter-efficient fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters, offer a promising solution by reducing the computational costs of tuning large language models (LLMs) while maintaining their performance. Existing studies have explored using PEFT and LLMs for various code-related tasks and found that the effectiveness of PEFT techniques is task-dependent. The state-of-the-art is limited to using LLMs with full fine-tuning to generate unit tests. The application of PEFT techniques in unit test generation remains underexplored. This paper investigates both full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt tuning, across thirteen models of different architectures and sizes. We use well-established benchmark datasets to evaluate their effectiveness in unit test generation and measure syntax correctness, CodeBLEU, pass@1, instruction coverage, branch coverage, and mutation score of the generated tests. Our findings show that LoRA can deliver performance comparable to full fine-tuning for unit test generation in several cases. If training costs are valued, prompt tuning is the most cost-effective approach, particularly for large models. However, the models tuned with full fine-tuning or PEFT may generate fewer executable test cases than the baseline model because they generate more tests calling nonexistent methods or having type mismatches. For the generated ones that are executable, the ones from the tuned models show better test coverage than those from the baseline model.

Paper Structure

This paper contains 43 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. PEFT Methods
  3. Additive
  4. Adapters
  5. Soft prompts
  6. Selective
  7. Reparametrization-based
  8. Related Work
  9. Unit test generation using LLMs
  10. PEFT methods on code-related tasks
  11. Experimental design
  12. Research Questions
  13. Fine-tuning methods selection
  14. Models
  15. Datasets
  16. ...and 28 more sections

Figures (2)

  • Figure 1: PEFT methods architecture illustration diagrams based on the reference implementation of each introductory paper. Striped modules are frozen. Magenta modules are trained.
  • Figure 4: Stacked bar chart of symmetrically normalized metrics value differences (increase or decrease) between baseline and tuned models, divided by the percentage of trainable parameters of models, on the Methods2Testrunnable dataset.