Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild
Aayush Kumar, Yasharth Bajpai, Sumit Gulwani, Gustavo Soares, Emerson Murphy-Hill
TL;DR
This study empirically examines how developers collaborate with in-IDE SWE agents during real open-source issue resolution, revealing that active, incremental collaboration and sharing of expert knowledge lead to higher success. It identifies barriers such as tacit knowledge gaps, unsolicited agent actions, and trust issues, advocating for agents that challenge developers and provide transparent, context-rich dialogue. The findings motivate design guidelines for SWE agents to support collaboration across all stages of the software development process, calibrate output scope, and avoid sycophantic behavior. Overall, the work demonstrates both the promise and the challenges of human-in-the-loop AI-assisted software development, offering concrete directions for more effective agent-enabled workflows.
Abstract
Software Engineering Agents (SWE agents) can autonomously perform development tasks on benchmarks like SWE Bench, but still face challenges when tackling complex and ambiguous real-world tasks. Consequently, SWE agents are often designed to allow interactivity with developers, enabling collaborative problem-solving. To understand how developers collaborate with SWE agents and the barriers they face in such interactions, we observed 19 developers using an in-IDE agent to resolve 33 open issues in repositories to which they had previously contributed. Participants successfully resolved about half of these issues, with those solving issues incrementally having greater success than those using a one-shot approach. Participants who actively collaborated with the agent and iterated on its outputs were also more successful, though they faced challenges in trusting the agent's responses and collaborating on debugging and testing. Our findings suggest that to facilitate successful collaborations, both SWE agents and developers should actively contribute to tasks throughout all stages of the software development process. SWE agents can enable this by challenging and engaging in discussions with developers, rather than being conclusive or sycophantic.