Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?

Vaishnavi Bhargava, Rajat Ghosh, Debojyoti Dutta

TL;DR

CPP-UT-Bench addresses the lack of real-world C++ unittest benchmarks by presenting a large, diverse dataset of 2,653 code–test pairs from 14 repositories across nine domains. It establishes three evaluation settings—few-shot in-context learning, parameter-efficient fine-tuning (LoRA), and full-parameter fine-tuning—and uses an LLM-as-a-Judge paradigm to assess code-generation quality. The study shows that fine-tuned models generally outperform base models, with average gains exceeding 70% in most experiments, while results vary by model and tuning approach. By releasing data, code, and fine-tuned weights, the work enables reproducible research and sets a framework for evaluating LLM-driven C++ unit-test generation in real-world codebases. The findings highlight the importance of context, model architecture, and tuning strategy for reliable test generation in C++.

Abstract

We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%.

CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?

TL;DR

CPP-UT-Bench addresses the lack of real-world C++ unittest benchmarks by presenting a large, diverse dataset of 2,653 code–test pairs from 14 repositories across nine domains. It establishes three evaluation settings—few-shot in-context learning, parameter-efficient fine-tuning (LoRA), and full-parameter fine-tuning—and uses an LLM-as-a-Judge paradigm to assess code-generation quality. The study shows that fine-tuned models generally outperform base models, with average gains exceeding 70% in most experiments, while results vary by model and tuning approach. By releasing data, code, and fine-tuned weights, the work enables reproducible research and sets a framework for evaluating LLM-driven C++ unit-test generation in real-world codebases. The findings highlight the importance of context, model architecture, and tuning strategy for reliable test generation in C++.

Abstract

We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%.

Paper Structure

This paper contains 19 sections, 4 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. CPP-UT-Bench
  3. Task Formulation
  4. Features of CPP-UT-Bench
  5. LLM-as-a-Judge
  6. Framework for C++ Unit Tests Generation
  7. Experiment Design
  8. Results
  9. Results for Few-Shot In-Context Learning (RQ-1)
  10. Results for Fine-Tuning (RQ-2)
  11. Mistral-7B-Instruct-v0.2
  12. TinyLlama
  13. CodeLlama
  14. Llama-3-8B
  15. Llama-3.1-8B
  16. ...and 4 more sections

Figures (13)

  • Figure 1: Data extraction pipeline for CPP-UT-Bench. It uses GitHub repos as upstream sources and then processes the code, unittest pairs extracted to create the benchmark dataset.
  • Figure 2: Data distribution of CPP-UT-Bench from 14 different GitHub Repositories. The dominant contribution (greater than $60 \%$) comes from Tensorflow and the least from PyTorch.
  • Figure 3: The diversity in {code, unit test} pairs in terms of line lengths across 14 different opensource repositories in CPP-UT-Bench.
  • Figure 4: Prompt for pairwise evaluation of two LLM generated responses (Assistant A and Assistant B) w.r.t. the ground truth.
  • Figure 5: Distribution of evaluation dataset for the few-shot in-context learning.
  • ...and 8 more figures