Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information

TL;DR

This paper addresses how contextual information about data distributions and AI performance affects human delegation decisions in human-AI teams. Using a between-subjects online experiment with四 treatments on an income-prediction task, it shows that only the combined data and AI contextual information significantly enhances human-AI team performance, mediated by instance-specific self-efficacy, AI efficacy, and perceived difficulty. The findings highlight the need for holistic contextual information in CSCW designs to optimize delegation and task allocation, while also noting cognitive load concerns when presenting AI-related information. Practically, the work provides guidance for onboarding and collaboration design and suggests avenues for integrating explanations and studying unstructured data in future research.

Abstract

The integration of artificial intelligence (AI) into human decision-making processes at the workplace presents both opportunities and challenges. One promising approach to leverage existing complementary capabilities is allowing humans to delegate individual instances of decision tasks to AI. However, enabling humans to delegate instances effectively requires them to assess several factors. One key factor is the analysis of both their own capabilities and those of the AI in the context of the given task. In this work, we conduct a behavioral study to explore the effects of providing contextual information to support this delegation decision. Specifically, we investigate how contextual information about the AI and the task domain influence humans' delegation decisions to an AI and their impact on the human-AI team performance. Our findings reveal that access to contextual information significantly improves human-AI team performance in delegation settings. Finally, we show that the delegation behavior changes with the different types of contextual information. Overall, this research advances the understanding of computer-supported, collaborative work and provides actionable insights for designing more effective collaborative systems.

Figures (8)

• Figure 1: In our human-AI delegation study, participants receive an instance (A) and two types of contextual information: (B) The distribution of a specific attribute within the data and (C) the AI's accuracy in relation to this attribute. This information supports participants in making informed decisions about whether to handle each instance manually or delegate it to the AI, optimizing the balance between human expertise and AI capabilities. During the study, the distributions of all features of an instance are available.
• Figure 2: The research model describes the relationship of the independent variable (contextual information), mediators (instance-specific self-efficacy, instance-specific AI efficacy, instance-specific difficulty perception) and the dependent variable (delegation performance).
• Figure 3: The design of the study is set up in four parts: first, participants are introduced to the task of the study, followed by part two in which they are randomly assigned to one of the four treatment groups. In part three, participants conduct the task of the study and have to fill out a questionnaire in part four.
• Figure 4: A comparison of the distribution of the mean performance per participant and 95% confidence intervals across the four treatments: control treatment in which participants do not receive contextual information, data treatment in which participants receive contextual data information, AI treatment in which participants receive contextual AI information and the data and AI treatment in which participants receive contextual data and AI information.
• Figure 5: A comparison of the distribution of the mean of the number of delegated instances per participant and 95% confidence intervals across the four treatments.