Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars

TL;DR

An incremental parser that allows early rejection of syntactically incorrect code, as well as efficient detection of complete programs for fill-in-the-middle (FIM) tasks, and the results demonstrate that constrained generation can significantly reduce the incidence of syntax errors in recommended code.

Abstract

Large Language Models are powerful tools for program synthesis and advanced auto-completion, but come with no guarantee that their output code is syntactically correct. This paper contributes an incremental parser that allows early rejection of syntactically incorrect code, as well as efficient detection of complete programs for fill-in-the-middle (FIM) tasks. We extend the Earley parsing algorithm to allow for left and right quotients of context-free grammars, and develop methods to handle quotienting of several context-sensitive features present in the grammars of many common programming languages. The result of these contributions is an efficient, general, and well-grounded method for left and right quotient parsing. To validate our theoretical contributions -- and the effectiveness of certain design decisions -- we evaluate our method on the particularly difficult case of FIM completion for Python 3, with syntax-correctness constraints. Our results demonstrate that constrained generation can significantly reduce the incidence of syntax errors in recommended code.

Figures (11)

• Figure 1: Conceptual overview of our method, integrated within a constrained generation process. Rounded rectangles represent data; cornered rectangles represent processes.
• Figure 2: Extract quotient language from CFG by traversing Earley item creation methods. For each chart $c_i$, and nonterminal $g \in V$, this algorithm computes the grammar representing how to complete any Earley items for $g$ that originate in $c_i$.
• Figure 3: We "parse" a regular language with a CFG, using the Earley algorithm modified to work over NFAs. The algorithm tracks how each Earley item is created by the scanner, predictor, and completer, shown as each Earley item are tracked, shown as thin solid, dotted, and dashed arrows respectively. When constructing the quotient grammar, each item in the final Earley chart corresponds to a production rule, representing the sequence necessary in order to complete each item. The algorithm finds all prediction arrows on the path from the initial state to the final state, and adds rules corresponding to each of those. Note that Earley items omit the "S $\rightarrow$", as there is only one nonterminal in this example.
• Figure 4: Example of lexer actions, and interaction with parser, for the string "ore(". Each box represents a lexer state.
• Figure 5: Incremental branch-based lexer and interaction with parser. $ST(c)$ returns the scannable terminals of chart $c$.

Theorems & Definitions (1)

• Definition 1