Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory

Cedric Faas, Sophie Kerstan, Richard Uth, Markus Langer, Anna Maria Feit

TL;DR

The paper investigates how to design oversight interfaces for domain experts supervising increasingly autonomous AI in high-stakes tasks. It employs four participatory co-design workshops with computer science and psychology experts overseeing AI-based grading, extracting user requirements and prototyped UI concepts. By integrating these findings with the SMART work design framework, the authors propose a twelve-point design framework that maps interface characteristics to psychological processes to support effective and meaningful oversight. The framework is argued to generalize beyond grading to other domains and to extend existing human–AI interaction guidelines by prioritizing engagement, autonomy, and relational needs in oversight tasks.

Abstract

As AI systems become increasingly capable and autonomous, domain experts' roles are shifting from performing tasks themselves to overseeing AI-generated outputs. Such oversight is critical, as undetected errors can have serious consequences or undermine the benefits of AI. Effective oversight, however, depends not only on detecting and correcting AI errors but also on the motivation and engagement of the oversight personnel and the meaningfulness they see in their work. Yet little is known about how domain experts approach and experience the oversight task and what should be considered to design effective and motivational interfaces that support human oversight. To address these questions, we conducted four co-design workshops with domain experts from psychology and computer science. We asked them to first oversee an AI-based grading system, and then discuss their experiences and needs during oversight. Finally, they collaboratively prototyped interfaces that could support them in their oversight task. Our thematic analysis revealed four key user requirements: understanding tasks and responsibilities, gaining insight into the AI's decision-making, contributing meaningfully to the process, and collaborating with peers and the AI. We integrated these empirical insights with the SMART model of work design to develop a generalizable framework of twelve design considerations. Our framework links interface characteristics and user requirements to the psychological processes underlying effective and satisfying work. Being grounded in work design theory, we expect these considerations to be applicable across domains and discuss how they extend existing guidelines for human-AI interaction and theoretical frameworks for effective human oversight by providing concrete guidance on the design of engaging and meaningful interfaces that support human oversight of AI systems.

Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory

TL;DR

The paper investigates how to design oversight interfaces for domain experts supervising increasingly autonomous AI in high-stakes tasks. It employs four participatory co-design workshops with computer science and psychology experts overseeing AI-based grading, extracting user requirements and prototyped UI concepts. By integrating these findings with the SMART work design framework, the authors propose a twelve-point design framework that maps interface characteristics to psychological processes to support effective and meaningful oversight. The framework is argued to generalize beyond grading to other domains and to extend existing human–AI interaction guidelines by prioritizing engagement, autonomy, and relational needs in oversight tasks.

Abstract

As AI systems become increasingly capable and autonomous, domain experts' roles are shifting from performing tasks themselves to overseeing AI-generated outputs. Such oversight is critical, as undetected errors can have serious consequences or undermine the benefits of AI. Effective oversight, however, depends not only on detecting and correcting AI errors but also on the motivation and engagement of the oversight personnel and the meaningfulness they see in their work. Yet little is known about how domain experts approach and experience the oversight task and what should be considered to design effective and motivational interfaces that support human oversight. To address these questions, we conducted four co-design workshops with domain experts from psychology and computer science. We asked them to first oversee an AI-based grading system, and then discuss their experiences and needs during oversight. Finally, they collaboratively prototyped interfaces that could support them in their oversight task. Our thematic analysis revealed four key user requirements: understanding tasks and responsibilities, gaining insight into the AI's decision-making, contributing meaningfully to the process, and collaborating with peers and the AI. We integrated these empirical insights with the SMART model of work design to develop a generalizable framework of twelve design considerations. Our framework links interface characteristics and user requirements to the psychological processes underlying effective and satisfying work. Being grounded in work design theory, we expect these considerations to be applicable across domains and discuss how they extend existing guidelines for human-AI interaction and theoretical frameworks for effective human oversight by providing concrete guidance on the design of engaging and meaningful interfaces that support human oversight of AI systems.
Paper Structure (45 sections, 5 figures, 2 tables)

This paper contains 45 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Human Oversight of AI Systems
  4. Psychological Processes at Work: Individual Needs and Motivation
  5. Human-centered AI
  6. Method
  7. Co-Design Workshop
  8. Participants & Collected Data
  9. Data Analysis Approach
  10. Results: Initial Approaches and Experiences of Domain experts
  11. Approaches
  12. Experiences
  13. Results: User Requirements and Supporting UI Elements
  14. Role Understanding: Clear understanding of one's role, the associated responsibilities, and tasks.
  15. AI Understanding: Clear understanding of AI capabilities and decision process.
  16. ...and 30 more sections

Figures (5)

  • Figure 1: Overview of the workshop design. We performed two sessions with each group, which both consisted of two tasks with different goals.
  • Figure 2: Overview of the main themes identified in our thematic analysis.
  • Figure 3: UI Elements that participants included to filter and group the student answers.
  • Figure 4: UI Elements that participants included to better understand the AI capabilities and its decision process.
  • Figure 5: UI Elements that participants included in their prototypes to meet their relational requirements.