JumpCoder: Go Beyond Autoregressive Coder via Online Modification

TL;DR

JumpCoder is introduced, a novel model-agnostic framework that enables human-like online modification and non-sequential generation to augment code LLMs and uses an Abstract Syntax Tree (AST) parser alongside the Generation Model Scoring to effectively judge the validity of each potential infill.

Abstract

While existing code large language models (code LLMs) exhibit impressive capabilities in code generation, their autoregressive sequential generation inherently lacks reversibility. This limitation hinders them from timely correcting previous missing statements during coding as humans do, often leading to error propagation and suboptimal performance. We introduce JumpCoder, a novel model-agnostic framework that enables human-like online modification and non-sequential generation to augment code LLMs. The key idea behind JumpCoder is to insert new code into the currently generated code when necessary during generation, which is achieved through an auxiliary infilling model that works in tandem with the code LLM. Since identifying the best infill position beforehand is intractable, we adopt an \textit{infill-first, judge-later} strategy, which experiments with filling at the $k$ most critical positions following the generation of each line, and uses an Abstract Syntax Tree (AST) parser alongside the Generation Model Scoring to effectively judge the validity of each potential infill. Extensive experiments using six state-of-the-art code LLMs across multiple and multilingual benchmarks consistently indicate significant improvements over all baselines. Our code is public at https://github.com/Keytoyze/JumpCoder.

Figures (17)

• Figure 1: An illustrative example demonstrating the difference between humans and LLMs. When a new variable is required, humans can jump back to the front section to define it, but LLMs, constrained by their autoregressive nature, can only continue generation and lead to error propagation.
• Figure 2: Schematic illustrations of traditional autoregressive coder and the proposed JumpCoder. Code lines are generated by generation model () and infilling model ().
• Figure 3: JumpCoder Framework. The iterative code update process comprises three important stages: Hybrid generation, Judging and Combination. Each iteration inserts a new line of code.
• Figure 4: Results on MultiPL-E. CL-I/P = CodeLlama-Instruct/Python. WC = WizardCoder-Python.
• Figure 5: Performance improvements of various baselines over greedy decoding on HumanEval.