Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding

Yifeng Di, Tianyi Zhang

TL;DR

The paper addresses misalignment between large language model (LLM) generated code and developer intent by introducing Programming with Interactive Grounding (PInG), which uses inline code comments as a bidirectional communication channel. The approach couples statement-level comment generation, human-edited feedback, and targeted code refinement, implemented as a VSCode extension with specialized models for commenting and refinement. Across benchmarks (HumanEval, MBPP) and a real-user study, PInG yields significant improvements in pass@1 and task success, demonstrating that interactive, fine-grained grounding can substantially boost code quality and developer trust. The work provides open-source code and data, highlighting practical gains in coding productivity and outlining avenues for interface improvements and broader language/model coverage.

Abstract

Large Language Models (LLMs) have demonstrated unprecedented capability in code generation. However, LLM-generated code is still plagued with a wide range of functional errors, especially for complex programming tasks that LLMs have not seen before. Recent studies have shown that developers often struggle with inspecting and fixing incorrect code generated by LLMs, diminishing their productivity and trust in LLM-based code generation. Inspired by the mutual grounding theory in communication, we propose an interactive approach that leverages code comments as a medium for developers and LLMs to establish a shared understanding. Our approach facilitates iterative grounding by interleaving code generation, inline comment generation, and contextualized user feedback through editable comments to align generated code with developer intent. We evaluated our approach on two popular benchmarks and demonstrated that our approach significantly improved multiple state-of-the-art LLMs, e.g., 17.1% pass@1 improvement for code-davinci-002 on HumanEval. Furthermore, we conducted a user study with 12 participants in comparison to two baselines: (1) interacting with GitHub Copilot, and (2) interacting with a multi-step code generation paradigm called Multi-Turn Program Synthesis. Participants completed the given programming tasks 16.7% faster and with 10.5% improvement in task success rate when using our approach. Both results show that interactively refining code comments enables the collaborative establishment of mutual grounding, leading to more accurate code generation and higher developer confidence.

Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding

TL;DR

The paper addresses misalignment between large language model (LLM) generated code and developer intent by introducing Programming with Interactive Grounding (PInG), which uses inline code comments as a bidirectional communication channel. The approach couples statement-level comment generation, human-edited feedback, and targeted code refinement, implemented as a VSCode extension with specialized models for commenting and refinement. Across benchmarks (HumanEval, MBPP) and a real-user study, PInG yields significant improvements in pass@1 and task success, demonstrating that interactive, fine-grained grounding can substantially boost code quality and developer trust. The work provides open-source code and data, highlighting practical gains in coding productivity and outlining avenues for interface improvements and broader language/model coverage.

Abstract

Large Language Models (LLMs) have demonstrated unprecedented capability in code generation. However, LLM-generated code is still plagued with a wide range of functional errors, especially for complex programming tasks that LLMs have not seen before. Recent studies have shown that developers often struggle with inspecting and fixing incorrect code generated by LLMs, diminishing their productivity and trust in LLM-based code generation. Inspired by the mutual grounding theory in communication, we propose an interactive approach that leverages code comments as a medium for developers and LLMs to establish a shared understanding. Our approach facilitates iterative grounding by interleaving code generation, inline comment generation, and contextualized user feedback through editable comments to align generated code with developer intent. We evaluated our approach on two popular benchmarks and demonstrated that our approach significantly improved multiple state-of-the-art LLMs, e.g., 17.1% pass@1 improvement for code-davinci-002 on HumanEval. Furthermore, we conducted a user study with 12 participants in comparison to two baselines: (1) interacting with GitHub Copilot, and (2) interacting with a multi-step code generation paradigm called Multi-Turn Program Synthesis. Participants completed the given programming tasks 16.7% faster and with 10.5% improvement in task success rate when using our approach. Both results show that interactively refining code comments enables the collaborative establishment of mutual grounding, leading to more accurate code generation and higher developer confidence.
Paper Structure (31 sections, 1 equation, 6 figures, 11 tables)

This paper contains 31 sections, 1 equation, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Approach
  4. Comment Generation
  5. Human Review
  6. Code Refinement
  7. Implementation Details
  8. Simulated User Study
  9. Benchmarks
  10. User Feedback Collection
  11. Evaluation Metrics
  12. Comparison Baselines
  13. Experiment Results
  14. RQ1: Effectiveness on Different Code Generation Models
  15. RQ2: Comparison to Other Prompting Approaches
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Generating a code solution in our pipeline.
  • Figure 2: Refining a code snippet with our approach via comment editing.
  • Figure 3: User Interface of PInG
  • Figure 4: The number of participants whose final code was functionally correct and the average number of iterations these participants took.
  • Figure 5: The distribution of participants’ confidence in the final code generated using the assigned tools.
  • ...and 1 more figures