Development and Validation of Engagement and Rapport Scales for Evaluating User Experience in Multimodal Dialogue Systems

TL;DR

The paper develops and validates two psychometric scales—engagement and rapport—to assess user experience quality in multimodal dialogue systems for English language learning. It builds 21 questionnaire items grounded in educational and social psychology and validates them using Cronbach's alpha and confirmatory factor analysis, comparing experiences with human tutors and a GPT-4–based AI agent. Results show clear multi-factor structures and higher engagement/rapport with human tutors, demonstrating the scales’ ability to differentiate interlocutor types. The work advances QoE evaluation for dialogue systems and points to future automatic estimation and broader domain applications to improve conversational agents.

Abstract

This study aimed to develop and validate two scales of engagement and rapport to evaluate the user experience quality with multimodal dialogue systems in the context of foreign language learning. The scales were designed based on theories of engagement in educational psychology, social psychology, and second language acquisition.Seventy-four Japanese learners of English completed roleplay and discussion tasks with trained human tutors and a dialog agent. After each dialogic task was completed, they responded to the scales of engagement and rapport. The validity and reliability of the scales were investigated through two analyses. We first conducted analysis of Cronbach's alpha coefficient and a series of confirmatory factor analyses to test the structural validity of the scales and the reliability of our designed items. We then compared the scores of engagement and rapport between the dialogue with human tutors and the one with a dialogue agent. The results revealed that our scales succeeded in capturing the difference in the dialogue experience quality between the human interlocutors and the dialogue agent from multiple perspectives.

Figures (7)

• Figure 1.1: System architecture
• Figure 1.2: Hypothesis model
• Figure 1.3: Baseline model
• Figure 1.4: Chrobach's alpha coefficient
• Figure 1.5: Path diagram and factor loadings (tutor)