Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the Interface: Redefining UX for Society-in-the-Loop AI Systems

Nahal Mafi, Sahar Maleki, Babak Rahimi Ardabili, Hamed Tabkhi

TL;DR

This paper argues that in AI-enabled HITL systems, UX must transcend frontend usability to encompass backend performance, organizational workflows, and decision making structures, and formalizes a new evaluative framework centered on four sociotechnical metrics.

Abstract

Artificial intelligence systems increasingly operate in decision-critical environments where probabilistic outputs and Human-in-the-Loop (HITL) interactions reshape user engagement. Traditional user experience (UX) frameworks, designed for deterministic systems, fail to capture these evolving sociotechnical dynamics. This paper argues that in AI-enabled HITL systems, UX must transcend frontend usability to encompass backend performance, organizational workflows, and decision making structures. We employ a mixed-methods approach, combining an inductive social construction analysis of 269 stakeholder insights with the deployment of an operational HITL video anomaly detection system. Our findings reveal that stakeholders experience AI through multifaceted themes: risk, governance, and organizational capacity. Experimental results further demonstrate how detection behavior and alert routing directly calibrate human oversight and workload. Grounded in these results, we formalize a new evaluative framework centered on four sociotechnical metrics: Accuracy (FPR/FNR), Operational Latency (response time), Adaptation Time (deployment burden), and Trust (validated automation scales). This framework redefines UX as a multi-layered construct spanning infrastructure and governance, providing a rigorous foundation for evaluating AI systems embedded within complex real-world ecosystems.

Beyond the Interface: Redefining UX for Society-in-the-Loop AI Systems

TL;DR

This paper argues that in AI-enabled HITL systems, UX must transcend frontend usability to encompass backend performance, organizational workflows, and decision making structures, and formalizes a new evaluative framework centered on four sociotechnical metrics.

Abstract

Artificial intelligence systems increasingly operate in decision-critical environments where probabilistic outputs and Human-in-the-Loop (HITL) interactions reshape user engagement. Traditional user experience (UX) frameworks, designed for deterministic systems, fail to capture these evolving sociotechnical dynamics. This paper argues that in AI-enabled HITL systems, UX must transcend frontend usability to encompass backend performance, organizational workflows, and decision making structures. We employ a mixed-methods approach, combining an inductive social construction analysis of 269 stakeholder insights with the deployment of an operational HITL video anomaly detection system. Our findings reveal that stakeholders experience AI through multifaceted themes: risk, governance, and organizational capacity. Experimental results further demonstrate how detection behavior and alert routing directly calibrate human oversight and workload. Grounded in these results, we formalize a new evaluative framework centered on four sociotechnical metrics: Accuracy (FPR/FNR), Operational Latency (response time), Adaptation Time (deployment burden), and Trust (validated automation scales). This framework redefines UX as a multi-layered construct spanning infrastructure and governance, providing a rigorous foundation for evaluating AI systems embedded within complex real-world ecosystems.
Paper Structure (18 sections, 7 figures, 1 table)

This paper contains 18 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Pre-AI vs. Post AI User Experience
  4. Social Construction
  5. Analytical Framework and Inductive Theme Development
  6. Decision Support and Analytical Value of AI Systems
  7. Risk Sensitivity, Errors, and False Positives in High-Stakes Contexts
  8. Trust, Transparency, Privacy, and Governance
  9. Human-in-the-Loop and Organizational Control
  10. Technology Experiment
  11. Human-In-The-Loop System Design
  12. Event-Based Anomaly Detection
  13. Defining New UX Metrics
  14. Accuracy
  15. Latency
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Comparison of pre-AI (left) and post-AI (right) UX design workflows. The pre-AI flow represents a linear, user-driven interaction model, while the post-AI flow introduces AI-mediated detection and alerting with human-in-the-loop feedback cycles (confirm, correct, comment), transforming UX into a dynamic co-adaptive process between users and intelligent systems.
  • Figure 2: Comparing the Pre-AI UX focuses and the Post-AI UX Impications.
  • Figure 3: Distribution of Coded Insights Across Themes
  • Figure 4: Theme Landscape by Decision Horizon and System Layer. Bubble size reflects theme frequency; color indicates category. Labels are automatically adjusted for clarity and connected to their bubbles.
  • Figure 5: End-to-end architecture of the proposed Human-in-the-Loop (HITL) system.
  • ...and 2 more figures