How to explain it to data scientists? A mixed-methods user study about explainable AI, using mental models for explanations

TL;DR

This mixed-methods study investigates explainable AI needs for data scientists within an industrial, semi-automated system that generates, deploys, and maintains ML-enabled capabilities. Through qualitative work to elicit initial mental-models for two key data-scientist tasks and a subsequent quantitative phase mapping explanation content to intents, the authors reveal that explanations must span application, system, and AI domains and can be organized sequentially or hierarchically as a causal story. Standardized explanation questions and iterative refinement of mental models improve coverage and perceived quality, with findings showing that content such as ranked combinations, interim results, and causal narratives most strongly support reason and trust intents. The work informs the design of explainable AI in production settings, highlighting domain-spanning content, structured storytelling, and task-specific guidance to support data scientists in selecting or maintaining ML-enabled components, while also noting limitations from participant reuse and suggesting avenues for broader usability validation.

Abstract

In the context of explainable artificial intelligence (XAI), limited research has identified role-specific explanation needs. This study investigates the explanation needs of data scientists, who are responsible for training, testing, deploying, and maintaining machine learning (ML) models in AI systems. The research aims to determine specific explanation content of data scientists. A task analysis identified user goals and proactive user tasks. Using explanation questions, task-specific explanation needs and content were identified. From these individual explanations, we developed a mental model for explanations, which was validated and revised through a qualitative study (n=12). In a second quantitative study (n=12), we examined which explanation intents (reason, comparison, accuracy, prediction, trust) require which type of explanation content from the mental model. The findings are: F1: Explanation content for data scientists comes from the application domain, system domain, and AI domain. F2: Explanation content can be complex and should be organized sequentially and/or in hierarchies (novelty claim). F3: Explanation content includes context, inputs, evidence, attributes, ranked list, interim results, efficacy principle, and input/output relationships (novelty claim). F4: Explanation content should be organized as a causal story. F5: Standardized explanation questions ensure complete coverage of explanation needs (novelty claim). F6: Refining mental models for explanations increases significantly its quality (novelty claim).

Figures (19)

• Figure 1: Research approach; step 3 and step 6 with the results are reported in this paper
• Figure 2: Revised mental model for user task "UT 1 Generate ML-model"
• Figure 3: Revised mental model for user task "UT 3 Maintain ML-model"
• Figure 4: Initial mental model for user task "UT 2 Generate ML-model"
• Figure 5: Initial mental model for user task "UT 3 Maintain ML-model"