Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fault-Aware Neural Code Rankers

Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, Jianfeng Gao

TL;DR

CodeRanker addresses the impracticality of executing generated code during inference by introducing a fault-aware neural ranker that predicts correctness without execution. By training on fine-grained fault categories derived from unit-test runs, it learns to distinguish correct versus faulted programs and to locate potential execution errors. The approach yields consistent improvements in pass@1 and exec@1 across multiple code-generation models and datasets (APPS, HumanEval, MBPP), and demonstrates transferability and useful ablations. While introducing inference-time overhead and not guaranteeing correctness, it offers a practical, execution-free method to boost functional outcomes in code generation tasks. This fault-aware ranking framework paves the way for more robust code synthesis in real-world development where executing arbitrary generated code is often unsafe or infeasible.

Abstract

Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets.

Fault-Aware Neural Code Rankers

TL;DR

CodeRanker addresses the impracticality of executing generated code during inference by introducing a fault-aware neural ranker that predicts correctness without execution. By training on fine-grained fault categories derived from unit-test runs, it learns to distinguish correct versus faulted programs and to locate potential execution errors. The approach yields consistent improvements in pass@1 and exec@1 across multiple code-generation models and datasets (APPS, HumanEval, MBPP), and demonstrates transferability and useful ablations. While introducing inference-time overhead and not guaranteeing correctness, it offers a practical, execution-free method to boost functional outcomes in code generation tasks. This fault-aware ranking framework paves the way for more robust code synthesis in real-world development where executing arbitrary generated code is often unsafe or infeasible.

Abstract

Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets.
Paper Structure (24 sections, 9 figures, 20 tables)

This paper contains 24 sections, 9 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Code generation
  4. Code ranking
  5. Fault-Aware Neural Code Ranker
  6. Code Ranker Dataset
  7. Code Ranker Tasks
  8. Code Ranker Models
  9. Evaluation
  10. Experiment Setup
  11. Main Results: CodeRanker improves code generation models
  12. Ablations
  13. Related works
  14. Conclusions and Future Directions
  15. Acknowledgements:
  16. ...and 9 more sections

Figures (9)

  • Figure 1: An example code generation task from the APPS dataset (train/3447).
  • Figure 2: Examples of execution errors.
  • Figure 3: Examples of intent errors.
  • Figure 4: Different classes of execution errors. See https://docs.python.org/3/library/exceptions.html for more details.
  • Figure 4: Training curves (smoothed) measuring ranked pass@1 over training steps for various rankers on different code generation models.
  • ...and 4 more figures