VizTrust: A Visual Analytics Tool for Capturing User Trust Dynamics in Human-AI Communication
Xin Wang, Stephanie Tulk Jesso, Sadamori Kojaku, David M Neyens, Min Sun Kim
TL;DR
VizTrust addresses the challenge of measuring dynamic user trust in real-time human–AI interactions by combining a front-end chat with a back-end multi-agent system that evaluates trust across $competence$, $integrity$, $benevolence$, and $predictability$, while tracking engagement, politeness, and emotional signals in a time-series dashboard. It leverages a cost-efficient Mixtral-8x7B LLM within a six-agent framework to perform per-turn trust evaluation and evidence gathering, enabling stakeholders to observe turning points and inform adaptive design. The system also analyzes user behavior signals through engagement metrics, politeness markers, and emotional tones, presenting these data in an interactive dashboard that supports quick design iterations. A case study demonstrates how VizTrust reveals trust declines linked to generic responses and misalignment with user needs, guiding targeted improvements to empathy and advice libraries, with broad implications for improving the safety and effectiveness of conversational agents.
Abstract
Trust plays a fundamental role in shaping the willingness of users to engage and collaborate with artificial intelligence (AI) systems. Yet, measuring user trust remains challenging due to its complex and dynamic nature. While traditional survey methods provide trust levels for long conversations, they fail to capture its dynamic evolution during ongoing interactions. Here, we present VizTrust, which addresses this challenge by introducing a real-time visual analytics tool that leverages a multi-agent collaboration system to capture and analyze user trust dynamics in human-agent communication. Built on established human-computer trust scales-competence, integrity, benevolence, and predictability-, VizTrust enables stakeholders to observe trust formation as it happens, identify patterns in trust development, and pinpoint specific interaction elements that influence trust. Our tool offers actionable insights into human-agent trust formation and evolution in real time through a dashboard, supporting the design of adaptive conversational agents that responds effectively to user trust signals.