Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches

Chen Zhu-Tian, Zeyu Xiong, Xiaoshuo Yao, Elena Glassman

TL;DR

The paper tackles the challenge of ineffective feedback during prompt construction for LLM-based code generation by introducing Language-Oriented Code Sketching, which incrementally derives a code sketch from the user's evolving prompt. The approach consists of Map, Assemble, and Preserve steps that map prompt phrases to code elements, assemble them into an AST-based sketch using linguistic relations, and preserve associations to support ongoing construction and guidance. This sketch serves both as an immediate preview of the intended structure and as a mechanism to steer subsequent LLM code generation, potentially reducing cognitive load and improving control over outputs. Preliminary feedback from a technology probe suggests the sketching approach can improve user experience, though robustness and disruption when accepting prompts remain as challenges to address in future work.

Abstract

Crafting effective prompts for code generation or editing with Large Language Models (LLMs) is not an easy task. Particularly, the absence of immediate, stable feedback during prompt crafting hinders effective interaction, as users are left to mentally imagine possible outcomes until the code is generated. In response, we introduce Language-Oriented Code Sketching, an interactive approach that provides instant, incremental feedback in the form of code sketches (i.e., incomplete code outlines) during prompt crafting. This approach converts a prompt into a code sketch by leveraging the inherent linguistic structures within the prompt and applying classic natural language processing techniques. The sketch then serves as an intermediate placeholder that not only previews the intended code structure but also guides the LLM towards the desired code, thereby enhancing human-LLM interaction. We conclude by discussing the approach's applicability and future plans.

Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches

TL;DR

The paper tackles the challenge of ineffective feedback during prompt construction for LLM-based code generation by introducing Language-Oriented Code Sketching, which incrementally derives a code sketch from the user's evolving prompt. The approach consists of Map, Assemble, and Preserve steps that map prompt phrases to code elements, assemble them into an AST-based sketch using linguistic relations, and preserve associations to support ongoing construction and guidance. This sketch serves both as an immediate preview of the intended structure and as a mechanism to steer subsequent LLM code generation, potentially reducing cognitive load and improving control over outputs. Preliminary feedback from a technology probe suggests the sketching approach can improve user experience, though robustness and disruption when accepting prompts remain as challenges to address in future work.

Abstract

Crafting effective prompts for code generation or editing with Large Language Models (LLMs) is not an easy task. Particularly, the absence of immediate, stable feedback during prompt crafting hinders effective interaction, as users are left to mentally imagine possible outcomes until the code is generated. In response, we introduce Language-Oriented Code Sketching, an interactive approach that provides instant, incremental feedback in the form of code sketches (i.e., incomplete code outlines) during prompt crafting. This approach converts a prompt into a code sketch by leveraging the inherent linguistic structures within the prompt and applying classic natural language processing techniques. The sketch then serves as an intermediate placeholder that not only previews the intended code structure but also guides the LLM towards the desired code, thereby enhancing human-LLM interaction. We conclude by discussing the approach's applicability and future plans.
Paper Structure (17 sections, 3 figures)

This paper contains 17 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-Oriented Code Sketching
  4. Map
  5. Independent Phrase: Matching Predefined Code Snippets or Existing Code
  6. Dependent Phrase: Inferring Based on Linguistic Features
  7. Assemble
  8. Preserve
  9. Guiding LLMs in Code Generation
  10. Preliminary User Feedback
  11. Implementation.
  12. User Feedback.
  13. Discussion and Future Work
  14. Target Contexts for Language-Oriented Code Sketching
  15. Effectiveness Depends on Phrase-Code Correspondence
  16. ...and 2 more sections

Figures (3)

  • Figure 1: As the user types a phrase, our system suggests corresponding code elements and previews their integration into the existing code. When the user accepts a suggestion, the code element is inserted and the association between the phrase and the code element is preserved.
  • Figure 2: Example rules for code assembly, each represented as a regular expression that specifies the linguistic relationship between the incoming phrase and an existing code-associated phrase within the dependency tree. The labels used (e.g, nsubj and dobj) are established terms in NLP.
  • Figure 3: Our approach is more effective with prompts (a) that include more code-relevant phrases, allowing for the creation of more detailed code sketches, in contrast to prompts (b) that contain fewer such phrases.