Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows

Jasmine Y. Shih, Vishal Mohanty, Yannis Katsis, Hariharan Subramonyam

TL;DR

This paper tackles the challenge of including domain experts in data science workflows by introducing CellSync, an integrated system that combines a Jupyter Notebook extension with an LLM-powered visualization dashboard. The Jupyter component tracks cell-level dataframe and model-metric changes and passes this information to a dashboard that renders interpretable summaries and visualizations, enabling domain experts to understand and discuss data operations without deep programming knowledge. The authors implement specific modules for natural language code descriptions, SnapGrid subset selection, and model-metrics extraction, and validate the approach through a preliminary 10-pair study, finding that the combined code summaries and visualizations promote stakeholder discussion and transparency, albeit with a learning curve. Overall, CellSync aims to reduce communication overhead and bridge domain expertise with data-science workflows, with potential for broader adoption in education and industry, as well as further refinement through domain-tuned prompting and extended deployments.

Abstract

Domain experts can play a crucial role in guiding data scientists to optimize machine learning models while ensuring contextual relevance for downstream use. However, in current workflows, such collaboration is challenging due to differing expertise, abstract documentation practices, and lack of access and visibility into low-level implementation artifacts. To address these challenges and enable domain expert participation, we introduce CellSync, a collaboration framework comprising (1) a Jupyter Notebook extension that continuously tracks changes to dataframes and model metrics and (2) a Large Language Model powered visualization dashboard that makes those changes interpretable to domain experts. Through CellSync's cell-level dataset visualization with code summaries, domain experts can interactively examine how individual data and modeling operations impact different data segments. The chat features enable data-centric conversations and targeted feedback to data scientists. Our preliminary evaluation shows that CellSync provides transparency and promotes critical discussions about the intents and implications of data operations.

Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows

TL;DR

This paper tackles the challenge of including domain experts in data science workflows by introducing CellSync, an integrated system that combines a Jupyter Notebook extension with an LLM-powered visualization dashboard. The Jupyter component tracks cell-level dataframe and model-metric changes and passes this information to a dashboard that renders interpretable summaries and visualizations, enabling domain experts to understand and discuss data operations without deep programming knowledge. The authors implement specific modules for natural language code descriptions, SnapGrid subset selection, and model-metrics extraction, and validate the approach through a preliminary 10-pair study, finding that the combined code summaries and visualizations promote stakeholder discussion and transparency, albeit with a learning curve. Overall, CellSync aims to reduce communication overhead and bridge domain expertise with data-science workflows, with potential for broader adoption in education and industry, as well as further refinement through domain-tuned prompting and extended deployments.

Abstract

Domain experts can play a crucial role in guiding data scientists to optimize machine learning models while ensuring contextual relevance for downstream use. However, in current workflows, such collaboration is challenging due to differing expertise, abstract documentation practices, and lack of access and visibility into low-level implementation artifacts. To address these challenges and enable domain expert participation, we introduce CellSync, a collaboration framework comprising (1) a Jupyter Notebook extension that continuously tracks changes to dataframes and model metrics and (2) a Large Language Model powered visualization dashboard that makes those changes interpretable to domain experts. Through CellSync's cell-level dataset visualization with code summaries, domain experts can interactively examine how individual data and modeling operations impact different data segments. The chat features enable data-centric conversations and targeted feedback to data scientists. Our preliminary evaluation shows that CellSync provides transparency and promotes critical discussions about the intents and implications of data operations.
Paper Structure (18 sections, 3 figures, 3 tables)

This paper contains 18 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. User Experience
  4. System Implementation
  5. Jupyter Notebook Extension
  6. Natural Language Code Description
  7. Computing Subset for SnapGrid
  8. Extracting Model Metrics
  9. Visualization Dashboard
  10. Preliminary Evaluation
  11. Findings
  12. Discussion and Conclusion
  13. Appendix
  14. LLM Prompts
  15. Column Relationship Extraction
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The CellSync visualization dashboard interface: (a) data version card containing code summary and SnapGrid along with column histograms, (b) clickable card navigator displaying a red dot indicating new comments, (c) detailed view for the selected column, (d) comments section for the current card, (e) bottom bar for switching between initial dataset table and data operation history, (f) dropdown menu for variable selection, (g) notification for new comments, and (h) collapsible legend for SnapGrid.
  • Figure 2: The CellSync data-scientist-facing chat interface rendered by the Jupyter Notebook extension.
  • Figure 3: (a) Text field for entering a natural language query to update the selected SnapGrid. (b) Cells with values that meet the query criteria are highlighted. (c) Table display of model metrics information.