How Much Automation Does a Data Scientist Want?

TL;DR

The study tackles whether practitioners want end-to-end AutoML by proposing a human-centered framework that encodes 5 automation levels, 10 lifecycle stages with 43 subtasks, 6 user roles, and 5 explanation types. It then empirically assesses needs via a large online survey of 217 DS/ML workers at a multinational IT company, revealing substantial demand for automation but a clear preference for human oversight in critical stages. Key findings show automation should be staged and role-specific rather than fully autonomous, with explainability (especially confidence and How/Why information) prioritized across workflows. The work advances a practical HITL AutoML design perspective and outlines steps for building user-controlled, explainable AutoML systems that better match diverse practitioner needs.

Abstract

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.

Figures (8)

• Figure 1: A 10 Stages, 43 Sub-tasks (not shown) DS/ML Lifecycle. This is a synthesized version from reviewing multiple scholarly publications and marketing reports aggarwal2019canwang2019humanai8020dilemmafeurer2015efficientberman2018realizinggartner2020magiclee2019humangil2019towards.
• Figure 2: Self-reported background information by respondents. Most respondents have 1-2 years of experience working as a professional, and in DS/ML projects.
• Figure 3: Participation rate of different roles across different stages in a data science project. Left: Different roles participate in different stages at a varying level. Right: The number of respondents corresponding to each role of the left graph.
• Figure 4: Comparison of respondents' current practice's level of automation and their desired levels of automation in each stage. The small bar chart at the top shows the number of responses collected for each stage.
• Figure 5: Preferences for automation of different roles for various stages. Blank columns indicate that we did not collect any data for that practitioner role at that stage.