Personality over Precision: Exploring the Influence of Human-Likeness on ChatGPT Use for Search

TL;DR

This study investigates how human-likeness in conversational search affects user trust and adoption, comparing ChatGPT with Google through an exploratory survey of 173 participants. It identifies two user segments—DUB and DUG—who diverge in trust, perceived human-likeness, and willingness to trade factuality for personalization and conversational flow. The findings highlight the central role of personalization and anthropomorphism in driving engagement, while also exposing overtrust risks, particularly for users who rely on ChatGPT as a starting point and then fact-check externally. These insights inform design strategies for conversational IR that balance engaging experiences with factual accuracy and user safety.

Abstract

Conversational search interfaces, like ChatGPT, offer an interactive, personalized, and engaging user experience compared to traditional search. On the downside, they are prone to cause overtrust issues where users rely on their responses even when they are incorrect. What aspects of the conversational interaction paradigm drive people to adopt it, and how it creates personalized experiences that lead to overtrust, is not clear. To understand the factors influencing the adoption of conversational interfaces, we conducted a survey with 173 participants. We examined user perceptions regarding trust, human-likeness (anthropomorphism), and design preferences between ChatGPT and Google. To better understand the overtrust phenomenon, we asked users about their willingness to trade off factuality for constructs like ease of use or human-likeness. Our analysis identified two distinct user groups: those who use both ChatGPT and Google daily (DUB), and those who primarily rely on Google (DUG). The DUB group exhibited higher trust in ChatGPT, perceiving it as more human-like, and expressed greater willingness to trade factual accuracy for enhanced personalization and conversational flow. Conversely, the DUG group showed lower trust toward ChatGPT but still appreciated aspects like ad-free experiences and responsive interactions. Demographic analysis further revealed nuanced patterns, with middle-aged adults using ChatGPT less frequently yet trusting it more, suggesting potential vulnerability to misinformation. Our findings contribute to understanding user segmentation, emphasizing the critical roles of personalization and human-likeness in conversational IR systems, and reveal important implications regarding users' willingness to compromise factual accuracy for more engaging interactions.

Figures (5)

• Figure 1: Distribution of people agreeing that ChatGPT has human-like qualities. DUB has higher agreement rates, but DUG also agrees more with the statement rather than disagreeing.
• Figure 2: Comparison between user groups for trade-off questions. DUB preferences don't differ significantly, while the DUG group agrees more with trading of factuality with human-likeness and conversationality, instead of ease-of-use.
• Figure 3: Inner group splits based on trust and perceived human-likeness. Most of the DUB group has high trust and high perceived human-likeness for ChatGPT, while the DUG group shows a split between high trust/human-likeness vs. low trust/human-likeness.
• Figure 4: Reasons people prefer ChatGPT over Google.
• Figure 5: Results of multiple-choice questions regarding which design aspects of ChatGPT/Google are liked over the other one. Looking at the numbers, ChatGPT had a clear advantage regarding design.