Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions

Jialin Li, Yuan Wu, Yi Chang

TL;DR

This work introduces ClarEval, a framework designed to assess an agent's "Collaborative Quotient" by simulating the inherent ambiguity of human communication, and proposes a metric suite led by Average Turns to Clarify and Key Question Coverage, which measure not just the correctness of the generated code, but the efficiency and precision of the collaboration.

Abstract

To integrate seamlessly into real-world software engineering, Code Agents must evolve from passive instruction followers into proactive collaborative partners. However, current evaluation paradigms predominantly reward "guessing" user intent under ideal conditions, neglecting the agent's ability to align with users through dialogue--a critical trait for collaborative intelligence. In this work, we propose a paradigm shift in evaluation to drive this transition. We introduce ClarEval, a framework designed to assess an agent's "Collaborative Quotient" by simulating the inherent ambiguity of human communication. By systematically injecting three types of realistic ambiguity (missing goals, premises, and ambiguous terminology) into standard tasks, we force agents to step out of their "generator" role and engage in requirement elicitation. To quantify this capability, we propose a metric suite led by Average Turns to Clarify (ATC) and Key Question Coverage (KQC), which measure not just the correctness of the generated code, but the efficiency and precision of the collaboration. Our experiments on eleven state-of-the-art agents reveal a stark reality: while models like GPT-5-Coder excel at coding, they often lack the strategic communication skills required for efficient partnership. ClarEval thus serves as a crucial roadmap for bridging the gap between strong coders and capable collaborators.The code is available at https://github.com/JialinLi13/ClarEval

ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions

TL;DR

This work introduces ClarEval, a framework designed to assess an agent's "Collaborative Quotient" by simulating the inherent ambiguity of human communication, and proposes a metric suite led by Average Turns to Clarify and Key Question Coverage, which measure not just the correctness of the generated code, but the efficiency and precision of the collaboration.

Abstract

To integrate seamlessly into real-world software engineering, Code Agents must evolve from passive instruction followers into proactive collaborative partners. However, current evaluation paradigms predominantly reward "guessing" user intent under ideal conditions, neglecting the agent's ability to align with users through dialogue--a critical trait for collaborative intelligence. In this work, we propose a paradigm shift in evaluation to drive this transition. We introduce ClarEval, a framework designed to assess an agent's "Collaborative Quotient" by simulating the inherent ambiguity of human communication. By systematically injecting three types of realistic ambiguity (missing goals, premises, and ambiguous terminology) into standard tasks, we force agents to step out of their "generator" role and engage in requirement elicitation. To quantify this capability, we propose a metric suite led by Average Turns to Clarify (ATC) and Key Question Coverage (KQC), which measure not just the correctness of the generated code, but the efficiency and precision of the collaboration. Our experiments on eleven state-of-the-art agents reveal a stark reality: while models like GPT-5-Coder excel at coding, they often lack the strategic communication skills required for efficient partnership. ClarEval thus serves as a crucial roadmap for bridging the gap between strong coders and capable collaborators.The code is available at https://github.com/JialinLi13/ClarEval
Paper Structure (56 sections, 8 equations, 3 figures, 14 tables)

This paper contains 56 sections, 8 equations, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Passive Code Generation vs. Real-World Ambiguity
  4. Agentic Frameworks and Repository Context
  5. Benchmarking Clarification and Efficiency
  6. Method
  7. Benchmark Construction
  8. Ambiguity Injection via Prompt Engineering
  9. Evaluation Settings
  10. Single-Turn Setting
  11. Multi-Turn Setting
  12. Human Verification Protocol
  13. Criteria Check.
  14. Ecological Validity Analysis.
  15. Consensus and Refinement.
  16. ...and 41 more sections

Figures (3)

  • Figure 1: Contrasting agent behaviors when given an underspecified instruction. This example compares a passive generation approach, which fails due to ambiguity, with a proactive clarification approach, which successfully resolves uncertainty through dialogue before generating the correct code, highlighting the critical role of clarification skills.
  • Figure 2: An overview for the ClarEval construction. First, original problems are obtained from the HumanEval dataset. Then, based on the three defined types of ambiguity—Missing Goal, Missing Premises, and Ambiguous Terminology—the original problems are transformed into ambiguous task descriptions using GPT-4o. Finally, the benchmark is refined and validated through human verification and annotation.
  • Figure 3: Performance Gap Analysis. Comparison of Pass@1 accuracy between the ambiguous baseline and the clarified upper bound across different ambiguity types. Ambiguous Terminology proves to be the most challenging scenario for passive code generation.