Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code

Jie JW Wu

TL;DR

This paper identifies a gap between LLM-assisted code generation and the standards of professional software engineering, arguing that effective communication—specifically, clarifying questions to resolve ambiguity—can raise the quality and trustworthiness of generated code. It proposes a communication-centered process combining a coder and an LLM-based communicator that detects low-confidence areas and solicits user input iteratively. A practical illustration with an example of computing the n-th Fibonacci number demonstrates how clarifying questions can refine implementation. The work lays out a path toward benchmarks and prompts that quantify and improve LLM communication skills, with the goal of narrowing the gap between AI coding assistants and top-tier developers.

Abstract

Large language models (LLMs) have significantly improved the ability to perform tasks in the field of code generation. However, there is still a gap between LLMs being capable coders and being top-tier software engineers. Based on the observation that toplevel software engineers often ask clarifying questions to reduce ambiguity in both requirements and coding solutions, I argue that the same should be applied to LLMs for code generation tasks. By asking probing questions in various topics before generating the final code, the challenges of programming with LLMs, such as unclear intent specification, lack of computational thinking, and undesired code quality, may be alleviated. This, in turn, increases confidence in the generated code. In this work, I explore how to leverage better communication skills to achieve greater confidence in generated code. I propose a communication-centered process that uses an LLM-generated communicator to identify issues with high ambiguity or low confidence in problem descriptions and generated code. I then ask clarifying questions to obtain responses from users for refining the code.

Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code

TL;DR

This paper identifies a gap between LLM-assisted code generation and the standards of professional software engineering, arguing that effective communication—specifically, clarifying questions to resolve ambiguity—can raise the quality and trustworthiness of generated code. It proposes a communication-centered process combining a coder and an LLM-based communicator that detects low-confidence areas and solicits user input iteratively. A practical illustration with an example of computing the n-th Fibonacci number demonstrates how clarifying questions can refine implementation. The work lays out a path toward benchmarks and prompts that quantify and improve LLM communication skills, with the goal of narrowing the gap between AI coding assistants and top-tier developers.

Abstract

Large language models (LLMs) have significantly improved the ability to perform tasks in the field of code generation. However, there is still a gap between LLMs being capable coders and being top-tier software engineers. Based on the observation that toplevel software engineers often ask clarifying questions to reduce ambiguity in both requirements and coding solutions, I argue that the same should be applied to LLMs for code generation tasks. By asking probing questions in various topics before generating the final code, the challenges of programming with LLMs, such as unclear intent specification, lack of computational thinking, and undesired code quality, may be alleviated. This, in turn, increases confidence in the generated code. In this work, I explore how to leverage better communication skills to achieve greater confidence in generated code. I propose a communication-centered process that uses an LLM-generated communicator to identify issues with high ambiguity or low confidence in problem descriptions and generated code. I then ask clarifying questions to obtain responses from users for refining the code.
Paper Structure (6 sections, 2 figures, 1 table)

This paper contains 6 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Communication-Centered Process
  3. Example Usage
  4. Background and Related Work
  5. Conclusion and Future Work
  6. Acknowledgements

Figures (2)

  • Figure 1: The visual illustration of the communication-centered process: the coder generates code according to the problem description and the communicator's information. The communicator detects the parts with low confidence where communication may help to get more information from the user. Then the communicator asks clarifying questions and gets the response. Finally, the response is sent back to the coder to refine the generated code.
  • Figure 2: An example of the communication-centered process.