Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study

Ian Drosos, Advait Sarkar, Xiaotong Xu, Carina Negreanu, Sean Rintel, Lev Tankelevitch

TL;DR

This paper investigates how generative AI can assist data analysis workflows by using participatory prompting with Bing Chat to study end users non expertly performing data sensemaking. The authors demonstrate that AI can streamline information foraging and support hypothesis generation and testing, while also introducing barriers in prompt formulation, contextualization, result volume, and verification. Through a qualitative study with fifteen participants, the work highlights design implications such as richer prompt scaffolding, better cross app integration, and explicit metacognitive support to manage verification and trust. The contributions include empirical insights into the opportunities and limits of AI assisted data sensemaking, a formal participatory prompting methodology, and guidance for future research and practical AI tool design that aligns with end user workflows.

Abstract

Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.

"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study

TL;DR

This paper investigates how generative AI can assist data analysis workflows by using participatory prompting with Bing Chat to study end users non expertly performing data sensemaking. The authors demonstrate that AI can streamline information foraging and support hypothesis generation and testing, while also introducing barriers in prompt formulation, contextualization, result volume, and verification. Through a qualitative study with fifteen participants, the work highlights design implications such as richer prompt scaffolding, better cross app integration, and explicit metacognitive support to manage verification and trust. The contributions include empirical insights into the opportunities and limits of AI assisted data sensemaking, a formal participatory prompting methodology, and guidance for future research and practical AI tool design that aligns with end user workflows.

Abstract

Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.
Paper Structure (53 sections, 3 figures, 2 tables)

This paper contains 53 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Sensemaking
  4. Definition of generative AI
  5. End-user programming
  6. Intelligent assistance for data analysis
  7. Method
  8. Participatory prompting
  9. Preparation
  10. Pilot
  11. Participatory prompting sessions
  12. Analysis method
  13. Results
  14. Overview of tasks
  15. Example turn-taking sessions
  16. ...and 38 more sections

Figures (3)

  • Figure 1: A simplified version of the data analyst's process, adapted from Pirolli and Card pirolli2005sensemaking. The process consists of (1) a foraging loop, in which the analyst transitions back and forth between a large set of data and a smaller set of interest through searching and filtering, and (2) a sensemaking loop, in which the analyst transitions back and forth between relevant data and hypotheses through hypothesis generation and testing of the hypotheses (analysis).
  • Figure 2: The workflow of iterative goal satisfaction with generative AI.
  • Figure 3: Bing Chat referencing UI (public design at the time of the study). The user message is in the dark blue bubble, top right. Bing Chat's response is in the white bubble, bottom left. Footnote-style superscripts indicate a supporting URL. The URLs are listed along the bottom of the chat response. A user can follow the superscripts or the links to read the web sources.