How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

TL;DR

This work characterize helpful planning suggestions and their impacts on analysts’ workflows, highlighting subtleties in contextual factors that impact suggestion helpfulness and design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.

Abstract

Data analysis is challenging as analysts must navigate nuanced decisions that may yield divergent conclusions. AI assistants have the potential to support analysts in planning their analyses, enabling more robust decision making. Though AI-based assistants that target code execution (e.g., Github Copilot) have received significant attention, limited research addresses assistance for both analysis execution and planning. In this work, we characterize helpful planning suggestions and their impacts on analysts' workflows. We first review the analysis planning literature and crowd-sourced analysis studies to categorize suggestion content. We then conduct a Wizard-of-Oz study (n=13) to observe analysts' preferences and reactions to planning assistance in a realistic scenario. Our findings highlight subtleties in contextual factors that impact suggestion helpfulness, emphasizing design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.

Figures (5)

• Figure 1: Planning assistance has limited support and is unexplored. Existing assistants, such as Github Copilot, are well-suited for execution assistance, such as suggesting data wrangling or modeling code.
• Figure 2: In our Wizard of Oz study, analysts work with a JupyterLab notebook containing the assistant interface (left). They are unaware of the existence of a human wizard, located in a separate room, operating the assistant backend (right). The wizard uses a general purpose LLM and integrates information prepared before the study. The wizard observes the analyst's notebook for opportunities for assistance and obtains the relevant notebook context (A). Next, the wizard either retrieves a prewritten suggestion developed prior to the study (B) or prompts the LLM (C) to generate a relevant suggestion (D). The wizard may further refine the suggestion (e.g., shorten its length) with additional prompting of the LLM (C-D Loop). Finally, after creating the suggestion, the wizard presents the suggestion back to the analyst (E). In this process, the wizard acts as the interface between the analyst and the LLM, deciding on the content and timing of the suggestion.
• Figure 3: Assistant User Interface. Our assistant is integrated into JupyterLab as a side panel. Analysts can open the side panel in the right sidebar (A) and turn the assistant on/off with a toggle (B). Suggestions are raised as cards in this side panel. When first turned on, the assistant welcomes the analyst with a startup message (C). When a new suggestion occurs, the side panel shows a loading animation of variable duration to call attention to it; variability in duration aims to indicate that the system is working and simulate a real system computing and generating a suggestion Fraser1991SimulatingSSRiek2012WizardOO. New suggestions also have a yellow highlight around them for greater visibility (D). Likewise, each suggestion has a corresponding context (D1) in the notebook, which is automatically scrolled into view when the suggestion is clicked (D2). Analysts can provide feedback for each suggestion via the thumbs-up and thumbs-down buttons (E). Finally, analysts can copy recommended code using the copy button (F). Note that the width of the side panel can be adjusted and is typically smaller than is pictured here for illustration.
• Figure 4: Analysts favor currently unsupported planning assistance. Analysts saw on average 11.85 (std=4.56) suggestions, 9.85 (std=4.16) of which were planning suggestions. All categories of intended planning suggestions (which may include code to help execute a suggestion) were found to be helpful by at least some analysts, highlighting the need for planning assistance that current assistants lack. For planning suggestions which analysts could reasonably incorporate into their notebooks (i.e., those that were not results interpretation and domain background), analysts integrated suggestions 51.6% of the time (47/91). Analyst reactions varied within each category, with responses ranging from finding the assistance helpful to neutral, or even unhelpful in certain instances. Our interviews and observations suggest that the effectiveness of a suggestion is not solely determined by its suggestion category but also depends on various nuanced factors (Sec. \ref{['sec:results']}). We provide full examples of the raised suggestions in the appendix.
• Figure 5: Study-informed Model of Analyst-Assistant Interaction Dynamics. Based on our study findings, we model the underlying influences and interactions between an analyst and an assistant. The assistant can receive information about the analyst's initiative (A) and the analysis contents (B) to develop an understanding of the analyst's goals and background as well as the current analysis plan and context (C). Note that the analyst's initiative is optional. This, in conjunction with the assistant's programmed goals (D) (e.g., offering more planning assistance or doing only what analysts request), informs the assistance (i.e., timing, organization, location, execution vs. planning, and contents) provided to analysts (E). After a suggestion is raised to the analyst, how it is perceived (F) is impacted by the analyst's background (G) and their current analysis plan (H). This then determines the extent to which the analyst accepts the suggestion in their planning (H) and/or execution (I) processes. The loop then continues as analysts update their analysis (B) or require additional assistance (A).