Statically Contextualizing Large Language Models with Typed Holes

TL;DR

The paper tackles semantic contextualization in LLM-based code completion by integrating language-server-derived type and binding information with hole-based program sketches in $Hazel$. It introduces static retrieval and static error correction through the Hazel Language Server, augmenting LM prompts with semantically relevant type definitions and headers, and validates the approach with MVUBench across Hazel and TypeScript. It also presents ChatLSP, a conservative extension to the Language Server Protocol, to expose static-contextualization capabilities to LLMs, and reports substantial gains in correctness, especially when type information is provided, across GPT-4 and StarCoder2. The work demonstrates that IDE-like static context can dramatically reduce hallucinations and improve token-efficient AI code assistance with broad applicability to both low-resource and high-resource languages.

Abstract

Large language models (LLMs) have reshaped the landscape of program synthesis. However, contemporary LLM-based code completion systems often hallucinate broken code because they lack appropriate context, particularly when working with definitions not in the training data nor near the cursor. This paper demonstrates that tight integration with the type and binding structure of a language, as exposed by its language server, can address this contextualization problem in a token-efficient manner. In short, we contend that AIs need IDEs, too! In particular, we integrate LLM code generation into the Hazel live program sketching environment. The Hazel Language Server identifies the type and typing context of the hole being filled, even in the presence of errors, ensuring that a meaningful program sketch is always available. This allows prompting with codebase-wide contextual information not lexically local to the cursor, nor necessarily in the same file, but that is likely to be semantically local to the developer's goal. Completions synthesized by the LLM are then iteratively refined via further dialog with the language server. To evaluate these techniques, we introduce MVUBench, a dataset of model-view-update (MVU) web applications. These applications serve as challenge problems due to their reliance on application-specific data structures. We find that contextualization with type definitions is particularly impactful. After introducing our ideas in the context of Hazel we duplicate our techniques and port MVUBench to TypeScript in order to validate the applicability of these methods to higher-resource languages. Finally, we outline ChatLSP, a conservative extension to the Language Server Protocol (LSP) that language servers can implement to expose capabilities that AI code completion systems of various designs can use to incorporate static context when generating prompts for an LLM.

Figures (19)

• Figure 1: Hazel Assistant Conversational Architecture (see \ref{['sec:acknowledgements']} for image acknowledgements)
• Figure 2: The Hazel IDE
• Figure 3: (a) The types relevant to the EmojiPainter MVU app. (b) An excerpt of already-implemented functions in another file (with definitions collapsed by Hazel's outliner). (c) The stubbed function header, where the developer has requested LLM completion by inserting in the hole.
• Figure 4: The Hazel Assistant defaults to providing only type-directed token completion.
• Figure 5: The Hazel LLM Assistant, combining static information with generative creation via language models, is capable of offering more substantial completions