Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks

Eitan Farchi, Shmulik Froimovich, Rami Katan, Orna Raz

TL;DR

This work introduces a methodology to generate and evaluate LaaJ implementations, utilizing an automatically generated benchmark, and develops an automated benchmark generation engine, which generates code in multiple programming languages for multiple code related tasks and serves as the input for LaaJ evaluation.

Abstract

LLMs can be used in a variety of code related tasks such as translating from one programming language to another, implementing natural language requirements and code summarization. Artifacts generated by state of the art LLM technology are expected to be useful in the sense that a user will be able to use the LLM generated artifact after a small number of easy modifications. Quantifying this vague notion is challenging and it is thus hard to determine the quality of code related LLM solutions. We refer to evaluation of LLM solutions using LLM judgment as "LLM as a Judge", or LaaJ for short. In this work we introduce a methodology to generate and evaluate LaaJ implementations, utilizing an automatically generated benchmark. The purpose of the benchmark is two fold, namely, it is used both to develop and validate the LaaJs and to validate and test the LLM code related solution using the LaaJs. To that end, we developed an automated benchmark generation engine, which generates code in multiple programming languages for multiple code related tasks and which serves as the input for LaaJ evaluation. We utilize a graph representation, G, of the potential code related generations. The graph vertices are generated artifacts and edges represent possible generations, e.g., the generation of a Java program from its natural language requirements. Utilizing a chain of LLM agents and G we generate code related artifacts. Using cycles in G we formulate expectations on the generated artifacts. Taking advantage of these formulated expectations enables the development and testing of reliable LLM judgement for usefulness of the artifacts generated by the solution. Our approach enables the creation of high quality code task solutions.

Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks

TL;DR

This work introduces a methodology to generate and evaluate LaaJ implementations, utilizing an automatically generated benchmark, and develops an automated benchmark generation engine, which generates code in multiple programming languages for multiple code related tasks and serves as the input for LaaJ evaluation.

Abstract

LLMs can be used in a variety of code related tasks such as translating from one programming language to another, implementing natural language requirements and code summarization. Artifacts generated by state of the art LLM technology are expected to be useful in the sense that a user will be able to use the LLM generated artifact after a small number of easy modifications. Quantifying this vague notion is challenging and it is thus hard to determine the quality of code related LLM solutions. We refer to evaluation of LLM solutions using LLM judgment as "LLM as a Judge", or LaaJ for short. In this work we introduce a methodology to generate and evaluate LaaJ implementations, utilizing an automatically generated benchmark. The purpose of the benchmark is two fold, namely, it is used both to develop and validate the LaaJs and to validate and test the LLM code related solution using the LaaJs. To that end, we developed an automated benchmark generation engine, which generates code in multiple programming languages for multiple code related tasks and which serves as the input for LaaJ evaluation. We utilize a graph representation, G, of the potential code related generations. The graph vertices are generated artifacts and edges represent possible generations, e.g., the generation of a Java program from its natural language requirements. Utilizing a chain of LLM agents and G we generate code related artifacts. Using cycles in G we formulate expectations on the generated artifacts. Taking advantage of these formulated expectations enables the development and testing of reliable LLM judgement for usefulness of the artifacts generated by the solution. Our approach enables the creation of high quality code task solutions.

Paper Structure

This paper contains 21 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Evaluation metrics for code LLM tasks
  3. The essence of our approach
  4. Quantifying the main requirement - usefulness of code related generated artifacts
  5. Creation of a self consistent benchmark
  6. Human provided seed topics for generation
  7. Seed code requirements
  8. The code task generation graph G
  9. Automatic generation
  10. Building and tuning the LLM based judge
  11. Architecture of the LaaJ
  12. Measuring and selecting the best judge
  13. A concrete LLM judgment evaluation example
  14. Experiment results: program idea clusters
  15. Pitfalls and how we address them
  16. ...and 6 more sections

Figures (1)

  • Figure 1: A flow of the self consistency benchmark generation for code generation.

Theorems & Definitions (2)

  • Example 1
  • Example 2