Blending Queries and Conversations: Understanding Tactics, Trust, Verification, and System Choice in Web Search and Chat Interactions

TL;DR

This study examines how users blend GenAI chat with traditional web search to address health information tasks, leveraging a concurrent think-aloud protocol with 22 participants. It reveals that GenAI chat is neither a universal fix nor a major regression compared to standard web search, with 78 tactics identified across five categories and trust-driven behaviours influencing outcomes. The findings show task type and user familiarity strongly shape interface usage, and that verification practices vary widely without consistently improving accuracy. The work highlights design and literacy challenges for AI-assisted information seeking and outlines open questions for improving verification, prompting, and user strategies in mixed-interface search systems.

Abstract

This paper presents a user study (N=22) where participants used an interface combining Web Search and a Generative AI-Chat feature to solve health-related information tasks. We study how people behaved with the interface, why they behaved in certain ways, and what the outcomes of these behaviours were. A think-aloud protocol captured their thought processes during searches. Our findings suggest that GenAI is neither a search panacea nor a major regression compared to standard Web Search interfaces. Qualitative and quantitative analyses identified 78 tactics across five categories and provided insight into how and why different interface features were used. We find evidence that pre-task confidence and trust both influenced which interface feature was used. In both systems, but particularly when using the chat feature, trust was often misplaced in favour of ease-of-use and seemingly perfect answers, leading to increased confidence post-search despite having incorrect results. We discuss what our findings mean in the context of our defined research questions and outline several open questions for future research.

Figures (7)

• Figure 1: Chat+Search Interface
• Figure 2: Pre- and post-search confidence levels for each task. Error bars show 95% confidence intervals.
• Figure 3: Pre- and post-search correctness percentages.
• Figure 4: Percentage of sessions using each approach.
• Figure 5: Number of tasks with post-search answers that were correct/incorrect grouped by search strategy used.