Do We Know What They Know We Know? Calibrating Student Trust in AI and Human Responses Through Mutual Theory of Mind

TL;DR

This paper investigates whether trust in AI- versus human-generated explanations translates into proportional reliance in educational settings, or whether trust and reliance are structurally decoupled. Using Mutual Theory of Mind, the authors conduct a qualitative, within-subject study with $N=8$ graduate CS students who evaluate AI-generated and human-generated explanations across Baseline, Construction, and Revision phases, totaling about $7.5$ hours of transcript data. The key finding is a systematic trust–reliance dissociation: students exhibit high trust in human experts yet low reliance due to fear of judgment, while they show low trust in AI but high reliance because of accessibility, anonymity, and a judgment-free environment. The study argues that trust is driven by epistemic judgments about competence, whereas reliance is driven by social affordances, implying that calibration efforts focused solely on accuracy or explainability may fail in educational contexts. Practically, the work advocates designing AI as a bridge to human help and equipping AI with transparency and scaffolding features to promote appropriate trust and reduce overreliance.

Abstract

Trust and reliance are often treated as coupled constructs in human-AI interaction research, with the assumption that calibrating trust will lead to appropriate reliance. We challenge this assumption in educational contexts, where students increasingly turn to AI for learning support. Through semi-structured interviews with graduate students (N=8) comparing AI-generated and human-generated responses, we find a systematic dissociation: students exhibit high trust but low reliance on human experts due to social barriers (fear of judgment, help-seeking anxiety), while showing low trust but high reliance on AI systems due to social affordances (accessibility, anonymity, judgment-free interaction). Using Mutual Theory of Mind as an analytical lens, we demonstrate that trust is shaped by epistemic evaluations while reliance is driven by social factors -- and these may operate independently.

Figures (1)

• Figure 1: Trust-Reliance Dissociation. Students exhibit an inverse relationship between trust and reliance when choosing between human experts and AI systems. Despite trusting human experts to provide accurate information, students often avoid seeking their help due to social anxiety and fear of judgment. Conversely, students readily rely on AI systems---despite potential hallucinations---because they offer 24/7 accessibility, anonymity, and a judgment-free environment where mistakes can be made privately.