Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering

Tao Dong, Harini Sampath, Ja Young Lee, Sherry Y. Shi, Andrew Macvean

TL;DR

The paper addresses the gap between current correctness-focused benchmarks and the needs of human-centered collaboration in software engineering AI agents. It introduces a foundational taxonomy of four agent-behavior expectations, derived from 91 enterprise agent-rule sets, and a Context-Adaptive Behavior (CAB) Framework with Time Horizon and Type of Work axes to adapt expectations across contexts. Through three studies—enterprise behavior taxonomy (Study 1), aspirational junior-developer behavior (Study 2), and prototyping-focused behavior (Study 3)—the work demonstrates both shared and context-specific expectations, highlighting the evolution from tool to partner in human-AI collaboration. The findings motivate a shift toward trajectory-based, behavioral evaluation and design guidance that emphasizes memory, communication, and metacognition to enable trustworthy, effective AI agents in real-world SWE workflows.

Abstract

As Large Language Models (LLMs) evolve from code generators into collaborative partners for software engineers, our methods for evaluation are lagging. Current benchmarks, focused on code correctness, fail to capture the nuanced, interactive behaviors essential for successful human-AI partnership. To bridge this evaluation gap, this paper makes two core contributions. First, we present a foundational taxonomy of desirable agent behaviors for enterprise software engineering, derived from an analysis of 91 sets of user-defined agent rules. This taxonomy defines four key expectations of agent behavior: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solving Problems Effectively, and Collaborating with the User. Second, recognizing that these expectations are not static, we introduce the Context-Adaptive Behavior (CAB) Framework. This emerging framework reveals how behavioral expectations shift along two empirically-derived axes: the Time Horizon (from immediate needs to future ideals), established through interviews with 15 expert engineers, and the Type of Work (from enterprise production to rapid prototyping, for example), identified through a prompt analysis of a prototyping agent. Together, these contributions offer a human-centered foundation for designing and evaluating the next generation of AI agents, moving the field's focus from the correctness of generated code toward the dynamics of true collaborative intelligence.

From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering

TL;DR

The paper addresses the gap between current correctness-focused benchmarks and the needs of human-centered collaboration in software engineering AI agents. It introduces a foundational taxonomy of four agent-behavior expectations, derived from 91 enterprise agent-rule sets, and a Context-Adaptive Behavior (CAB) Framework with Time Horizon and Type of Work axes to adapt expectations across contexts. Through three studies—enterprise behavior taxonomy (Study 1), aspirational junior-developer behavior (Study 2), and prototyping-focused behavior (Study 3)—the work demonstrates both shared and context-specific expectations, highlighting the evolution from tool to partner in human-AI collaboration. The findings motivate a shift toward trajectory-based, behavioral evaluation and design guidance that emphasizes memory, communication, and metacognition to enable trustworthy, effective AI agents in real-world SWE workflows.

Abstract

As Large Language Models (LLMs) evolve from code generators into collaborative partners for software engineers, our methods for evaluation are lagging. Current benchmarks, focused on code correctness, fail to capture the nuanced, interactive behaviors essential for successful human-AI partnership. To bridge this evaluation gap, this paper makes two core contributions. First, we present a foundational taxonomy of desirable agent behaviors for enterprise software engineering, derived from an analysis of 91 sets of user-defined agent rules. This taxonomy defines four key expectations of agent behavior: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solving Problems Effectively, and Collaborating with the User. Second, recognizing that these expectations are not static, we introduce the Context-Adaptive Behavior (CAB) Framework. This emerging framework reveals how behavioral expectations shift along two empirically-derived axes: the Time Horizon (from immediate needs to future ideals), established through interviews with 15 expert engineers, and the Type of Work (from enterprise production to rapid prototyping, for example), identified through a prompt analysis of a prototyping agent. Together, these contributions offer a human-centered foundation for designing and evaluating the next generation of AI agents, moving the field's focus from the correctness of generated code toward the dynamics of true collaborative intelligence.
Paper Structure (37 sections, 2 figures, 1 table)

This paper contains 37 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating SWE Agents
  4. Evaluating AI’s Ability to Collaborate with Users
  5. Understanding Skills of Software Engineers
  6. Study 1: Expected Agent Behavior in Enterprise Software Engineering
  7. Findings
  8. Adhering to Standards and Processes
  9. Ensure Code Quality and Reliability
  10. Solve Problems Effectively
  11. Collaborate with the User
  12. Interplay between Expectations of Agent Behavior
  13. Study 2: Aspirational Agent Behavior at the Junior Developer Level
  14. Methods
  15. Findings
  16. ...and 22 more sections

Figures (2)

  • Figure 1: The majority of the 15 developers in our interview study had more than 6 years of professional experience, and the majority use AI agents for their professional work every day. In addition, the majority of our participants supervised or mentored junior software developers multiple times in their respective careers.
  • Figure 2: The Context-Adaptive Behavior (CAB) Framework extends the taxonomy of desirable agent behaviors developed for an enterprise setting along two empirically-derived axes. The Time Horizon axis maps the evolution from current needs to future ideals. The Type of Work axis reveals how expectations change when shifting from production-grade engineering to exploratory development. Text in green with a plus sign indicates expected behaviors unique in its quadrant. Text in gray with a downward arrow represents an expectation less prominent compared to the “Enterprise Production” base line.