Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Code Generation: Assessing Code LLM Maturity with Postconditions

Fusen He, Juan Zhai, Minxue Pan

TL;DR

This paper proposes a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities, and augments the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models.

Abstract

Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: https://github.com/MatureModel/PostcondGen

Beyond Code Generation: Assessing Code LLM Maturity with Postconditions

TL;DR

This paper proposes a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities, and augments the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models.

Abstract

Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: https://github.com/MatureModel/PostcondGen
Paper Structure (19 sections, 3 figures, 10 tables)

This paper contains 19 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Postcondition & Taxonomy
  4. Methodology
  5. Maturity Model Design
  6. Prompt Generation
  7. Few-short Generation
  8. Category-based Generation
  9. Results
  10. Experimental setups
  11. Correctness of LLM-generated Postconditions
  12. LLMs Capabilities on Generating Different Types of Postconditions
  13. Completeness of LLM-generated Postconditions
  14. Impacts of Individual Postconditioin Types
  15. Conclusion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An Example for Postconditions Generated by LLMs
  • Figure 2: Overview of Prompt Generation
  • Figure 3: Prompt template for our approach. Underline text would be replaced by concrete contents specific to the programming language, targeted program and postcondition category that we are handling. Italicized text would only be included in category-specific prompts.