Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation

Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, Shing-Chi Cheung

TL;DR

A code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL are proposed, providing directions for future research improvements.

Abstract

Code benchmarks such as HumanEval are widely adopted to evaluate the capabilities of Large Language Models (LLMs), providing insights into their strengths and weaknesses. However, current benchmarks primarily exercise LLMs' capability on common coding tasks (e.g., bubble sort, greatest common divisor), leaving domain-specific coding tasks (e.g., computation, system, cryptography) unexplored. To fill this gap, we propose a multi-domain code benchmark, DOMAINEVAL, designed to evaluate LLMs' coding capabilities thoroughly. Our pipeline works in a fully automated manner, enabling a push-bottom construction from code repositories into formatted subjects under study. Interesting findings are observed by evaluating 12 representative LLMs against DOMAINEVAL. We notice that LLMs are generally good at computation tasks while falling short on cryptography and system coding tasks. The performance gap can be as much as 68.94% (80.94% - 12.0%) in some LLMs. We also observe that generating more samples can increase the overall performance of LLMs, while the domain bias may even increase. The contributions of this study include a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL, providing directions for future research improvements. The leaderboard is available at https://domaineval.github.io/.

DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation

TL;DR

A code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL are proposed, providing directions for future research improvements.

Abstract

Code benchmarks such as HumanEval are widely adopted to evaluate the capabilities of Large Language Models (LLMs), providing insights into their strengths and weaknesses. However, current benchmarks primarily exercise LLMs' capability on common coding tasks (e.g., bubble sort, greatest common divisor), leaving domain-specific coding tasks (e.g., computation, system, cryptography) unexplored. To fill this gap, we propose a multi-domain code benchmark, DOMAINEVAL, designed to evaluate LLMs' coding capabilities thoroughly. Our pipeline works in a fully automated manner, enabling a push-bottom construction from code repositories into formatted subjects under study. Interesting findings are observed by evaluating 12 representative LLMs against DOMAINEVAL. We notice that LLMs are generally good at computation tasks while falling short on cryptography and system coding tasks. The performance gap can be as much as 68.94% (80.94% - 12.0%) in some LLMs. We also observe that generating more samples can increase the overall performance of LLMs, while the domain bias may even increase. The contributions of this study include a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL, providing directions for future research improvements. The leaderboard is available at https://domaineval.github.io/.
Paper Structure (19 sections, 8 figures, 2 tables)

This paper contains 19 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Benchmark Construction
  3. Domain Repository Collection
  4. Test-Method Matching & Selection
  5. Test-Method Matching
  6. Test-Method Selection
  7. Instruction Generation
  8. Benchmark Statistics
  9. Experiments
  10. Experiment Setup
  11. Overall Result
  12. Domain Biases
  13. LLMs Biases
  14. Impact of Generated Samples
  15. Case Study
  16. ...and 4 more sections

Figures (8)

  • Figure 1: Domain-specific Code from DomainEval
  • Figure 2: Pipeline of DomainEval Construction and the Domains, Repositories and Statistics in DomainEval.
  • Figure 3: A example subject in DomainEval. Each subject consists of three components: instruction for LLM evaluation, reference solution, and a series of tests.
  • Figure 4: Instruction Generation prompt for LLM
  • Figure 5: Distribution of line counts for the reference code (context included) across domains in DomainEval.
  • ...and 3 more figures