On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review

TL;DR

The paper addresses the lack of a shared trust definition in SE for AI assisted software development. It conducts a cross disciplinary literature review and a critical SE analysis to demonstrate that SE often equates trust with artifact acceptance, lacking integration with established trust theories. It argues for adopting models from psychology and philosophy (notably Mayer 1995 and related work) and provides concrete recommendations to embed trust concepts and instruments into SE studies. The work advocates for mature interdisciplinary trust research in SE to enable calibrated trust in AI assisted software development with implications for design, evaluation, and governance.

Abstract

Trust is a fundamental concept in human decision-making and collaboration that has long been studied in philosophy and psychology. However, software engineering (SE) articles often use the term trust informally; providing an explicit definition or embedding results in established trust models is rare. In SE research on AI assistants, this practice culminates in equating trust with the likelihood of accepting generated content, which, in isolation, does not capture the full conceptual complexity of trust. Without a common definition, true secondary research on trust is impossible. The objectives of our research were: (1) to present the psychological and philosophical foundations of human trust, (2) to systematically study how trust is conceptualized in SE and the related disciplines human-computer interaction and information systems, and (3) to discuss limitations of equating trust with content acceptance, outlining how SE research can adopt existing trust models to overcome the widespread informal use of the term trust. We conducted a literature review across disciplines and a critical review of recent SE articles with a focus on trust conceptualizations. We found that trust is rarely defined or conceptualized in SE articles. Related disciplines commonly embed their methodology and results in established trust models, clearly distinguishing, for example, between initial trust and trust formation and between appropriate and inappropriate trust. On a meta-scientific level, other disciplines even discuss whether and when trust can be applied to AI assistants at all. Our study reveals a significant maturity gap of trust research in SE compared to other disciplines. We provide concrete recommendations on how SE researchers can adopt established trust models and instruments to study trust in AI assistants beyond the acceptance of generated software artifacts.

Figures (3)

• Figure 1: Trust model by Mayer1995 applied to AI assistants with references to existing measurement instruments MayerDavies1999DBLP:journals/isr/McKnightCK02Frazier01102013Scholz17012025DavidSchoorman02012016gillespie201520DBLP:journals/jds/SchmidtBT20.
• Figure 2: Trust model by Mayer1995 (own illustration based on Mayer1995).
• Figure 3: Trust model by Lee2004Trust (own illustration based on Lee2004Trust).