Data Discovery using LLMs -- A Study of Data User Behaviour
Christin Katharina Kreutz, Anja Perry, Tanja Friedrich
TL;DR
This paper tackles the challenge of data discovery by evaluating whether large language models can effectively support researchers in expressing data needs. It presents a controlled user study with 32 researchers performing unguided and persona-guided data-search tasks using ChatGPT and Perplexity, combining think-aloud protocols, interviews, and UX measures. The findings indicate that participants treat LLMs as tools rather than equal conversational partners, with persona prompting altering interaction dynamics but not overall satisfaction for non-experts; experienced users show some UX benefits under persona prompting. The work provides empirical guidance for designing LLM-enabled data-search tools and highlights considerations for human–AI collaboration, reliability, and domain-specific adoption.
Abstract
Data search for scientific research is more complex than a simple web search. The emergence of large language models (LLMs) and their applicability for scientific tasks offers new opportunities for researchers who are looking for data, e.g., to freely express their data needs instead of fitting them into restrictions of data catalogues and portals. However, this also creates uncertainty about whether LLMs are suitable for this task. To answer this question, we conducted a user study with 32 researchers. We qualitatively and quantitively analysed participants' information interaction behaviour while searching for data using LLMs in two data search tasks, one in which we prompted the LLM to behave as a persona. We found that participants interact with LLMs in natural language, but LLMs remain a tool for them rather than an equal conversational partner. This changes slightly when the LLM is prompted to behave as a persona, but the prompting only affects participants' user experience when they are already experienced in LLM use.