Trustworthy AI Software Engineers

TL;DR

The paper tackles how AI coding agents can be reframed as trustworthy software engineers operating within socio-technical human-AI SE teams. It defines an AI software engineer as an AI agent capable of handling SE tasks across the lifecycle, planning with tools, reasoning over context, producing artefacts beyond code, and collaborating with humans under ethical constraints. It contributes a multidimensional trustworthiness framework—spanning technical quality, transparency, epistemic humility, and societal alignment—while acknowledging evaluation challenges and a trust-measurement gap. The authors advocate ethics-by-design and continuous, context-aware assessment to guide the design, evaluation, and governance of future human-AI SE systems.

Abstract

With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we re-examine what it means for an AI agent to be considered a software engineer and then critically think about what makes such an agent trustworthy. \textit{Grounded} in established definitions of software engineering (SE) and informed by recent research on agentic AI systems, we conceptualise AI software engineers as participants in human-AI SE teams composed of human software engineers and AI models and tools, and we distinguish trustworthiness as a key property of these systems and actors rather than a subjective human attitude. Based on historical perspectives and emerging visions, we identify key dimensions that contribute to the trustworthiness of AI software engineers, spanning technical quality, transparency and accountability, epistemic humility, and societal and ethical alignment. We further discuss how trustworthiness can be evaluated and demonstrated, highlighting a fundamental trust measurement gap: not everything that matters for trust can be easily measured. Finally, we outline implications for the design, evaluation, and governance of AI SE systems, advocating for an ethics-by-design approach to enable appropriate trust in future human-AI SE teams.