Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, Xin Xia

TL;DR

This paper proposes a framework, namely $\boldsymbol{\mathcal{R}}\mathbf{Eval}$, for evaluating code reasoning abilities and consistency of code LLMs with program execution, and utilizes existing code benchmarks and adapt them to new benchmarks within this framework.

Abstract

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

TL;DR

This paper proposes a framework, namely , for evaluating code reasoning abilities and consistency of code LLMs with program execution, and utilizes existing code benchmarks and adapt them to new benchmarks within this framework.

Abstract

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.
Paper Structure (36 sections, 7 equations, 5 figures, 4 tables)

This paper contains 36 sections, 7 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Code Execution and Reasoning
  4. Code Execution Behavior
  5. Code Reasoning with Large Language Models
  6. Consistency for Large Language Models
  7. Code LLMs and Benchmarks
  8. Code LLMs
  9. Benchmarks
  10. REval Framework
  11. Overview of Framework
  12. Runtime Behavior Reasoning
  13. Code Coverage Prediction (CCP)
  14. Program State Prediction (PSP)
  15. Execution Path Prediction (EPP)
  16. ...and 21 more sections

Figures (5)

  • Figure 1: The demonstration of code reasoning tasks in CRUXEval gu2024cruxeval and $\bm{\mathcal{R}}$Eval. "IC": Incremental Consistency.
  • Figure 2: Overview of our framework. Benchmark Construction: we adapt the base benchmarks to fit our framework by execution. Runtime Behavior Reasoning: we propose four tasks including CCP, PSP, EPP, and OP, which challenge LLMs to perform code reasoning. Incremental Consistency Evaluation: we evaluate if the model can maintain consistency on sequentially related tasks (i.e., Incremental Consistency).
  • Figure 3: The prompt template for our empirical study. Note that the "Thoughts" part is used only we leverage Chain-of-Thought (CoT) wei2022chain prompting.
  • Figure 4: Average Accuracy of Runtime Behavior Reasoning and Incremental Consistency Score for different models, sorted in descending order according to average accuracy.
  • Figure 5: A tricky problem of EPP from HumanEval/59 codex. The prediction of GPT-4-Turbo is correct and the other two models (i.e., GPT-3.5-Turbo and CodeLlama-34B-Instruct) fail to finish it. The problem description is simplified for a concise presentation.