Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Unseen Code Tests In Infinitum

Marcel Zalmanovici, Orna Raz, Eitan Farchi, Iftach Freund

TL;DR

The paper tackles the challenge of evaluating code-generation capabilities of language-driven systems while avoiding data leakage from benchmarks. It proposes an AST-based benchmark-generation framework that generalizes across tasks and programming languages, exemplified by the auto-regression low-level text-to-code benchmark and an accompanying debugging dictionary for regression analysis. Key contributions include a generalizable benchmark construction method from existing code with unit tests, ground-truth runnable code, and AST-derived statistics to identify error-prone constructs, plus a debugging dictionary to facilitate systematic failure analysis. Initial results on a NAPS-derived tiny/small dataset provide anecdotal evidence of usefulness, particularly in tracing improvements and regression across model versions. The approach aims to support leakage-resilient benchmarking and ongoing regression testing in code-centric generation scenarios, with potential extension to more languages and repositories.

Abstract

Large Language Models (LLMs) are used for many tasks, including those related to coding. An important aspect of being able to utilize LLMs is the ability to assess their fitness for specific usages. The common practice is to evaluate LLMs against a set of benchmarks. While benchmarks provide a sound foundation for evaluation and comparison of alternatives, they suffer from the well-known weakness of leaking into the training data \cite{Xu2024Benchmarking}. We present a method for creating benchmark variations that generalize across coding tasks and programming languages, and may also be applied to in-house code bases. Our approach enables ongoing generation of test-data thus mitigating the leaking into the training data issue. We implement one benchmark, called \textit{auto-regression}, for the task of text-to-code generation in Python. Auto-regression is specifically created to aid in debugging and in tracking model generation changes as part of the LLM regression testing process.

Generating Unseen Code Tests In Infinitum

TL;DR

The paper tackles the challenge of evaluating code-generation capabilities of language-driven systems while avoiding data leakage from benchmarks. It proposes an AST-based benchmark-generation framework that generalizes across tasks and programming languages, exemplified by the auto-regression low-level text-to-code benchmark and an accompanying debugging dictionary for regression analysis. Key contributions include a generalizable benchmark construction method from existing code with unit tests, ground-truth runnable code, and AST-derived statistics to identify error-prone constructs, plus a debugging dictionary to facilitate systematic failure analysis. Initial results on a NAPS-derived tiny/small dataset provide anecdotal evidence of usefulness, particularly in tracing improvements and regression across model versions. The approach aims to support leakage-resilient benchmarking and ongoing regression testing in code-centric generation scenarios, with potential extension to more languages and repositories.

Abstract

Large Language Models (LLMs) are used for many tasks, including those related to coding. An important aspect of being able to utilize LLMs is the ability to assess their fitness for specific usages. The common practice is to evaluate LLMs against a set of benchmarks. While benchmarks provide a sound foundation for evaluation and comparison of alternatives, they suffer from the well-known weakness of leaking into the training data \cite{Xu2024Benchmarking}. We present a method for creating benchmark variations that generalize across coding tasks and programming languages, and may also be applied to in-house code bases. Our approach enables ongoing generation of test-data thus mitigating the leaking into the training data issue. We implement one benchmark, called \textit{auto-regression}, for the task of text-to-code generation in Python. Auto-regression is specifically created to aid in debugging and in tracking model generation changes as part of the LLM regression testing process.
Paper Structure (9 sections, 1 figure, 3 tables)

This paper contains 9 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. AST-based benchmark generation
  3. Data
  4. Metrics
  5. Debugging dictionary
  6. Benchmark results
  7. Conclusions
  8. Limitations
  9. Appendix A: Code Examples

Figures (1)

  • Figure 1: The process of generating custom benchmarks and running them