To Trust or Distrust Trust Measures: Validating Questionnaires for Trust in AI

TL;DR

This study provides the first comprehensive psychometric validation of the Trust between People and Automation (TPA) scale in AI contexts, testing whether trust and distrust should be modeled as separate constructs. Using a preregistered 2x2 online experiment (N = 1485) with trustworthy vs untrustworthy AI in two scenarios (autonomous vehicle and chatbot), the authors show that a two-factor TPA model fits data well ($ ext{RMSEA}=0.103$, $ ext{CFI}=0.952$, $ ext{SRMR}=0.051$) and yields reliable trust ($ ext{α}=0.94$, $ ext{ω}=0.95$) and distrust ($ ext{α}=0.88$, $ ext{ω}=0.89$) scores. They find robust criterion validity: higher trust scores for trustworthy AI and higher distrust scores for untrustworthy AI, along with distinct patterns of convergent/divergent validity involving situational trust and affect. The results advocate for measuring both trust and distrust in human-AI research and recommend adopting a two-dimensional framing to better capture calibrated and warranted trust. This work advances standardized measurement in AI trust research and has practical implications for XAI design and evaluation, enabling more nuanced assessments and comparisons across studies.

Abstract

Despite the importance of trust in human-AI interactions, researchers must adopt questionnaires from other disciplines that lack validation in the AI context. Motivated by the need for reliable and valid measures, we investigated the psychometric quality of two trust questionnaires, the Trust between People and Automation scale (TPA) by Jian et al. (2000) and the Trust Scale for the AI Context (TAI) by Hoffman et al. (2023). In a pre-registered online experiment (N = 1485), participants observed interactions with trustworthy and untrustworthy AI (autonomous vehicle and chatbot). Results support the psychometric quality of the TAI while revealing opportunities to improve the TPA, which we outline in our recommendations for using the two questionnaires. Furthermore, our findings provide additional empirical evidence of trust and distrust as two distinct constructs that may coexist independently. Building on our findings, we highlight the opportunities and added value of measuring both trust and distrust in human-AI research and advocate for further work on both constructs.

Figures (2)

• Figure 1: An illustration of the 2x2 online experiment stimuli by condition (trustworthy vs. untrustworthy) x scenario (chatbot vs. automated vehicle), constituting four groups in total.
• Figure 2: Conceptual frameworks of trust and distrust. (a) the one-dimensional conceptualization places trust on a single continuum ranging from low to high trust (adapted from castelfranchi2010trust). (b) the two-dimensional conceptualization of trust and distrust separates trust and distrust scores into two distinct dimensions. Quadrant I: high trust, high distrust. Quadrant II: low trust, high distrust. Quadrant III: low trust, low distrust. Quadrant IV: high trust, low distrust (adapted from lewicki1998trust).