Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Treefix: Enabling Execution with a Tree of Prefixes

Beatriz Souza, Michael Pradel

TL;DR

Treefix introduces a three-step, LLM-driven framework that constructs a tree of code prefixes to enable execution of incomplete Python snippets. By statically identifying undefined references, iteratively refining prefixes via runtime errors, and targeting uncovered lines, it achieves substantially higher line coverage than state-of-the-art baselines, with open-source and Stack Overflow snippets reaching up to $84\%$ and $82\%$ coverage respectively. The approach produces a large, diverse set of concrete values (over $16{,}000$ unique values across $1{,}462$ snippets) and demonstrates practical effectiveness while maintaining manageable prefix sets and execution costs. The results show that each step contributes meaningfully, dependency resolution via pipreqs is important, and the method enables multiple execution paths, which is beneficial for dynamic analysis tasks. Treefix thus offers a scalable, open-source path toward robust execution of arbitrary code snippets in environments with missing context and dependencies, with data and prompts available for replication.

Abstract

The ability to execute code is a prerequisite for various dynamic program analyses. Learning-guided execution has been proposed as an approach to enable the execution of arbitrary code snippets by letting a neural model predict likely values for any missing variables. Although state-of-the-art learning-guided execution approaches, such as LExecutor, can enable the execution of a relative high amount of code, they are limited to predicting a restricted set of possible values and do not use any feedback from previous executions to execute even more code. This paper presents Treefix, a novel learning-guided execution approach that leverages LLMs to iteratively create code prefixes that enable the execution of a given code snippet. The approach addresses the problem in a multi-step fashion, where each step uses feedback about the code snippet and its execution to instruct an LLM to improve a previously generated prefix. This process iteratively creates a tree of prefixes, a subset of which is returned to the user as prefixes that maximize the number of executed lines in the code snippet. In our experiments with two datasets of Python code snippets, Treefix achieves 25% and 7% more coverage relative to the current state of the art in learning-guided execution, covering a total of 84% and 82% of all lines in the code snippets.

Treefix: Enabling Execution with a Tree of Prefixes

TL;DR

Treefix introduces a three-step, LLM-driven framework that constructs a tree of code prefixes to enable execution of incomplete Python snippets. By statically identifying undefined references, iteratively refining prefixes via runtime errors, and targeting uncovered lines, it achieves substantially higher line coverage than state-of-the-art baselines, with open-source and Stack Overflow snippets reaching up to and coverage respectively. The approach produces a large, diverse set of concrete values (over unique values across snippets) and demonstrates practical effectiveness while maintaining manageable prefix sets and execution costs. The results show that each step contributes meaningfully, dependency resolution via pipreqs is important, and the method enables multiple execution paths, which is beneficial for dynamic analysis tasks. Treefix thus offers a scalable, open-source path toward robust execution of arbitrary code snippets in environments with missing context and dependencies, with data and prompts available for replication.

Abstract

The ability to execute code is a prerequisite for various dynamic program analyses. Learning-guided execution has been proposed as an approach to enable the execution of arbitrary code snippets by letting a neural model predict likely values for any missing variables. Although state-of-the-art learning-guided execution approaches, such as LExecutor, can enable the execution of a relative high amount of code, they are limited to predicting a restricted set of possible values and do not use any feedback from previous executions to execute even more code. This paper presents Treefix, a novel learning-guided execution approach that leverages LLMs to iteratively create code prefixes that enable the execution of a given code snippet. The approach addresses the problem in a multi-step fashion, where each step uses feedback about the code snippet and its execution to instruct an LLM to improve a previously generated prefix. This process iteratively creates a tree of prefixes, a subset of which is returned to the user as prefixes that maximize the number of executed lines in the code snippet. In our experiments with two datasets of Python code snippets, Treefix achieves 25% and 7% more coverage relative to the current state of the art in learning-guided execution, covering a total of 84% and 82% of all lines in the code snippets.
Paper Structure (33 sections, 9 figures, 6 tables, 1 algorithm)

This paper contains 33 sections, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Approach
  4. Problem Statement
  5. Overview and Running Example
  6. Main Algorithm
  7. Step 1: Undefinedness Guidance
  8. Step 2: Error Guidance
  9. Step 3: Coverage Guidance
  10. Execution and Coverage Measurement
  11. Installing Third-Party Dependencies
  12. Post-Processing of Prefixes
  13. Measure Coverage
  14. Evaluation
  15. Experimental Setup
  16. ...and 18 more sections

Figures (9)

  • Figure 1: Example of code to execute and predicted values.
  • Figure 2: Tree of prefixes generated in different steps of Treefix. Prefixes highlighted in green are added to $P$. The best prefix $p_{best}$ is highlighted with a star.
  • Figure 3: Prompt for undefinedness guidance.
  • Figure 4: Response specification.
  • Figure 5: Prompt for error guidance.
  • ...and 4 more figures