Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis

Parm Suksakul, Nathan Kittichaikoonkij, Nakhin Polthai, Aung Pyae

TL;DR

Through five-cycle thematic analysis of 1,435 codewords, four themes are derived: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination.

Abstract

Developing and deploying AI applications in organizations is challenging when human decision authority and oversight are underspecified across the system lifecycle. Although Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged, operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented. We report a multi-source qualitative study: a retrospective diary study of a customer-support chatbot and semi-structured interviews with eight AI experts from academia and industry. Through five-cycle thematic analysis of 1,435 codewords, we derive four themes: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination. These themes provide empirical inputs for subsequent HITL framework design and validation.

Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis

TL;DR

Through five-cycle thematic analysis of 1,435 codewords, four themes are derived: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination.

Abstract

Developing and deploying AI applications in organizations is challenging when human decision authority and oversight are underspecified across the system lifecycle. Although Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged, operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented. We report a multi-source qualitative study: a retrospective diary study of a customer-support chatbot and semi-structured interviews with eight AI experts from academia and industry. Through five-cycle thematic analysis of 1,435 codewords, we derive four themes: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination. These themes provide empirical inputs for subsequent HITL framework design and validation.
Paper Structure (14 sections, 4 figures)

This paper contains 14 sections, 4 figures.

Table of Contents

  1. Introduction
  2. BACKGROUND
  3. METHODOLOGY
  4. Study Design and Methodological Rationale
  5. Data Source 1: Retrospective Diary Study and Case Description
  6. Data Source 2: AI Expert Interview Study
  7. Preparation and Analytical Approach
  8. RESULTS
  9. Theme 1: AI Governance and Human Authority
  10. Theme 2: Human-in-the-Loop Iterative Refinement
  11. Theme 3: AI System Lifecycle and Operational Constraints
  12. Theme 4: Human–AI Team Collaboration and Coordination
  13. Discussion and Future Work
  14. Conclusion

Figures (4)

  • Figure 1: Overview of the customer support chatbot workflow–business AI application case study
  • Figure 2: Open coding workflow
  • Figure 3: Thematic analysis process
  • Figure 4: Hierarchical relationship of four high-level themes