Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SelfPiCo: Self-Guided Partial Code Execution with LLMs

Zhipeng Xue, Zhipeng Gao, Shaohua Wang, Xing Hu, Xin Xia, Shanping Li

TL;DR

SelfPiCo introduces an LLM-in-the-loop framework to dynamically guide the execution of non-executable partial code. By integrating a Runtime Engine, an Interactive Value Predictor, and a Complementary Type Predictor within a LITL loop, it iteratively injects meaningful values to make partial Python code executable. Empirical results show substantial improvements in code coverage and runtime type-error detection over Lexecutor baselines, including on Stack Overflow snippets, with robust ablation and sensitivity analyses. The work demonstrates practical impact for dynamic analysis and debugging of incomplete code, while suggesting avenues to broaden domain information and scalability in future work.

Abstract

Code executability plays a vital role in software debugging and testing (e.g., detecting runtime exceptions or assertion violations). However, code execution, especially partial or arbitrary code execution, is a non-trivial task due to missing definitions and complex third-party dependencies. To make partial code (such as code snippets posted on the web or code fragments deep inside complex software projects) executable, the existing study has proposed a machine learning model to predict the undefined element types and inject the pre-defined dummy values into execution. However, the performance of their tool is limited due to its simply designed dummy values and the inability to continue learning. In this paper, we design and implement a novel framework, named SelfPiCo (Self Guided Partial Code Executor), to dynamically guide partial code execution by incorporating the open-source LLM (i.e., Code Llama) within an interactive loop. Particularly, SelfPiCo leverages few-shot in-context learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning based on fine-tuning the Code Llama model. SelfPiCo continuously learns from code execution results and refines its predictions step after step. Our evaluations demonstrate that SelfPiCo can execute 72.7% and 83.3% of all lines in the open-source code and Stack Overflow snippets, outperforming the most recent state-of-the-art Lexecutor by 37.9% and 33.5%, respectively. Moreover, SelfPiCo successfully detected 18 and 33 runtime type error issues by executing the partial code from eight GitHub software projects and 43 Stack Overflow posts, demonstrating the practical usage and potential application of our framework in practice.

SelfPiCo: Self-Guided Partial Code Execution with LLMs

TL;DR

SelfPiCo introduces an LLM-in-the-loop framework to dynamically guide the execution of non-executable partial code. By integrating a Runtime Engine, an Interactive Value Predictor, and a Complementary Type Predictor within a LITL loop, it iteratively injects meaningful values to make partial Python code executable. Empirical results show substantial improvements in code coverage and runtime type-error detection over Lexecutor baselines, including on Stack Overflow snippets, with robust ablation and sensitivity analyses. The work demonstrates practical impact for dynamic analysis and debugging of incomplete code, while suggesting avenues to broaden domain information and scalability in future work.

Abstract

Code executability plays a vital role in software debugging and testing (e.g., detecting runtime exceptions or assertion violations). However, code execution, especially partial or arbitrary code execution, is a non-trivial task due to missing definitions and complex third-party dependencies. To make partial code (such as code snippets posted on the web or code fragments deep inside complex software projects) executable, the existing study has proposed a machine learning model to predict the undefined element types and inject the pre-defined dummy values into execution. However, the performance of their tool is limited due to its simply designed dummy values and the inability to continue learning. In this paper, we design and implement a novel framework, named SelfPiCo (Self Guided Partial Code Executor), to dynamically guide partial code execution by incorporating the open-source LLM (i.e., Code Llama) within an interactive loop. Particularly, SelfPiCo leverages few-shot in-context learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning based on fine-tuning the Code Llama model. SelfPiCo continuously learns from code execution results and refines its predictions step after step. Our evaluations demonstrate that SelfPiCo can execute 72.7% and 83.3% of all lines in the open-source code and Stack Overflow snippets, outperforming the most recent state-of-the-art Lexecutor by 37.9% and 33.5%, respectively. Moreover, SelfPiCo successfully detected 18 and 33 runtime type error issues by executing the partial code from eight GitHub software projects and 43 Stack Overflow posts, demonstrating the practical usage and potential application of our framework in practice.
Paper Structure (26 sections, 5 figures, 7 tables, 1 algorithm)

This paper contains 26 sections, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Our Approach
  4. Runtime Engine
  5. Interactive Value Predictor
  6. Interactive Value Generator
  7. Execution Value Checker
  8. LITL (LLMs-In-The-Loop) Algorithm
  9. Complementary Type Predictor
  10. Pre-defined Dummy Value
  11. Prompt Engineering
  12. Implementation
  13. Evaluation
  14. Experimental Setup
  15. RQ1: Effectiveness of SelfPiCo
  16. ...and 11 more sections

Figures (5)

  • Figure 1: A Type Error Detected From Partial Code
  • Figure 2: The Overview of SelfPiCo
  • Figure 3: Venn Graph for Fully Executed Code Snippets Reported by Our Approach and Lexecutor, Open-Source Project Functions (left) and Stack Overflow Code Snippets (right)
  • Figure 4: Performance of Interactive Value Predictor
  • Figure 5: Type error detected by SelfPiCo