"I Like Sunnie More Than I Expected!": Exploring User Expectation and Perception of an Anthropomorphic LLM-based Conversational Agent for Well-Being Support

TL;DR

Results showed that user engagement was high with both systems, and both systems exceeded users' expectations along the utility dimension, highlighting AI's potential to offer useful intervention activity recommendations.

Abstract

The human-computer interaction (HCI) research community has a longstanding interest in exploring the mismatch between users' actual experiences and expectation toward new technologies, for instance, large language models (LLMs). In this study, we compared users' (N = 38) initial expectations against their post-interaction perceptions of two LLM-powered mental well-being intervention activity recommendation systems. Both systems have a built-in LLM to recommend a personalized well-being intervention activity, but one system (Sunnie) has an anthropomorphic conversational interaction design via elements such as appearance, persona, and natural conversation. Results showed that user engagement was high with both systems, and both systems exceeded users' expectations along the utility dimension, highlighting AI's potential to offer useful intervention activity recommendations. In addition, Sunnie further outperformed the non-anthropomorphic baseline system in relational warmth. These findings suggest that anthropomorphic conversational interaction design may be particularly effective in fostering warmth in mental health support contexts.

Figures (7)

• Figure 1: The system architecture of Sunnie. Sunnie is supported by an LLM with a meticulously designed prompt framework. A user interacts with Sunnie through a series of interfaces, including selecting and typing their feelings with buttons and text input, communicating with Sunnie in multi-turn conversation, receiving personalized activity recommendations, and deciding whether to take the activity.
• Figure 2: The anthropomorphic design of Sunnie in the conversation interface is highlighted with red circles. These designs include the title of "Chat with Sunnie," the anthropomorphic appearance of Sunnie as a conversational agent, and the design of a "Sunnie is typing..." animation while waiting for the generated response from GPT-4.
• Figure 3: The user interaction flow with Sunnie includes six stages: 1) user selects the emoji that best captures how they are currently feeling, 2) user selects one or more keyword(s) to describe their feelings, 3) user describes the perceived source of their feelings and optionally uploads an image, 4) Sunnie initiates a brief multi-turn conversation for personalized well-being coaching, 5) Sunnie provides personalized activity recommendation, and 6) user determines whether to engage in the activity.
• Figure 4: The prompting framework for Sunnie comprises four modules: Sunnie's persona, conversation protocol, system setting, and response optimization.
• Figure 5: The 3-day participatory study design overview. We had 38 participants in total (20 in one group and 18 in the other). Each group 1) completed a pre-study survey, 2) interacted with both the Baseline and the Sunnie system in alternating order, and 3) completed a post-study survey. On the third day, four participants volunteered for a semi-structured post-study interview.