Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysis

TL;DR

Domain-specific data analysis often suffers from gaps in organizational concepts and terminology, limiting LLM usefulness. The authors present DomainDA, an autonomous agent that uses domain-specific exemplars retrieved from domain logs and a stepwise in-context pseudo-code workflow to iteratively generate, execute, and refine analyses with minimal human input. They contribute a 45-item DomainQuery dataset, demonstrate improved code correctness and richer interpretations with domain-specific exemplars, and explore two design probes (OHA and SLA) through interviews with nine data scientists to understand user perceptions and workflow integration. The work offers a practical path toward domain-aware, collaborative LLM-assisted data analysis and emphasizes explainability and cross-user collaboration in specialized domains.

Abstract

Despite demonstrating robust capabilities in performing tasks related to general-domain data-operation tasks, Large Language Models (LLMs) may exhibit shortcomings when applied to domain-specific tasks. We consider the design of domain-specific AI-powered data analysis tools from two dimensions: interaction and user agency. We implemented two design probes that fall on the two ends of the two dimensions: an open-ended high agency (OHA) prototype and a structured low agency (SLA) prototype. We conducted an interview study with nine data scientists to investigate (1) how users perceived the LLM outputs for data analysis assistance, and (2) how the two test design probes, OHA and SLA, affected user behavior, performance, and perceptions. Our study revealed insights regarding participants' interactions with LLMs, how they perceived the results, and their desire for explainability concerning LLM outputs, along with a noted need for collaboration with other users, and how they envisioned the utility of LLMs in their workflow.

Figures (4)

• Figure 1: Preliminary study interface.
• Figure 2: Preliminary study interface.
• Figure 3: Preliminary study interface.
• Figure 4: ChatGPT-like Prototype used as design probe in the interview study.