Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes

TL;DR

This study investigates enriching qualitative data collection in online surveys by embedding four theory-based interview probes (Descriptive, Idiographic, Clarifying, Explanatory) into an LLM-powered chatbot. Using a $N=64$ split-plot design across three HCI interview stages (Exploration, Requirements, Evaluation), it assesses impact on response quality via Gricean Maxims and on user experience. Findings show Idiographic (and Descriptive) probes yield higher quality insights, with stage-specific strengths, while Explanatory can be more repetitive; user experience is generally consistent across probes. The work offers practical recommendations, methodological insights, and open-source chatbot implementations to advance qualitative data collection in surveys.

Abstract

Surveys are a widespread method for collecting data at scale, but their rigid structure often limits the depth of qualitative insights obtained. While interviews naturally yield richer responses, they are challenging to conduct across diverse locations and large participant pools. To partially bridge this gap, we investigate the potential of using LLM-based chatbots to support qualitative data collection through interview probes embedded in surveys. We assess four theory-based interview probes: descriptive, idiographic, clarifying, and explanatory. Through a split-plot study design (N=64), we compare the probes' impact on response quality and user experience across three key stages of HCI research: exploration, requirements gathering, and evaluation. Our results show that probes facilitate the collection of high-quality survey data, with specific probes proving effective at different research stages. We contribute practical and methodological implications for using chatbots as research tools to enrich qualitative data collection.

Figures (4)

• Figure 1: Chatbot interface with a conversation from our study in exploration stage with the idiographic interview probe.
• Figure 1: Visualisation of the interview probe main effects for each quality measure.
• Figure 2: Visualisation of the interview stage main effects for each quality measure.
• Figure 3: Visualisation of the interaction effects between interview stage and interview probe main for each quality measure.