Unpacking Human-AI interactions: From interaction primitives to a design space

TL;DR

The paper tackles the challenge of designing meaningful Human-AI interactions by proposing a semi-formal design space built on interaction primitives and message-passing patterns. It introduces a running robot-emotion-learning example to demonstrate low-level primitives (provide, request) and information types (input, output, feedback) that are then composed into higher-level interaction patterns. The approach integrates XAI, HITL, and hybrid intelligence paradigms, and aligns with existing guidelines by offering a design-materials framework that can be prototyped and extended. The contribution is a practical, extendable language for describing and prototyping HAI interactions, enabling designers and practitioners to explore new interaction possibilities while considering implementation concerns and sociotechnical issues.

Abstract

This paper aims to develop a semi-formal design space for Human-AI interactions, by building a set of interaction primitives which specify the communication between users and AI systems during their interaction. We show how these primitives can be combined into a set of interaction patterns which can provide an abstract specification for exchanging messages between humans and AI/ML models to carry out purposeful interactions. The motivation behind this is twofold: firstly, to provide a compact generalisation of existing practices, that highlights the similarities and differences between systems in terms of their interaction behaviours; and secondly, to support the creation of new systems, in particular by opening the space of possibilities for interactions with models. We present a short literature review on frameworks, guidelines and taxonomies related to the design and implementation of HAI interactions, including human-in-the-loop, explainable AI, as well as hybrid intelligence and collaborative learning approaches. From the literature review, we define a vocabulary for describing information exchanges in terms of providing and requesting particular model-specific data types. Based on this vocabulary, a message passing model for interactions between humans and models is presented, which we demonstrate can account for existing systems and approaches. Finally, we build this into design patterns as mid-level constructs that capture common interactional structures. We discuss how this approach can be used towards a design space for Human-AI interactions that creates new possibilities for designs as well as keeping track of implementation issues and concerns.

Figures (17)

• Figure 1: Proposed Approach. We review existing guidelines, taxonomies and frameworks related to the design and implementation of human-AI interactions. Our goal is to unpack HAI interactions into a set of interaction primitives and patterns, based on which more complex interactions can be defined. Taking into consideration both technical and design challenges, we describe how the proposed primitives can be used to define a design space which can inform both design and implementation choices for HAI interactions.
• Figure 2: Interactive robot learning yu2019interactive using descriptions of interaction patterns and primitives. The image shows the original interaction schema as a box and arrow diagram, followed by a high level description of the HAI interactions patterns, the unpacking into actions involving the exchange of information between the human and the system, and a description of the interaction requirements.
• Figure 3: Communication types during human-model interactions. AI models provide/receive information in a model-specific format: (a) input (test/train data, relations, hyperparameters, etc.), (c) output (labels, predictions, estimated parameters, projections, etc.) and (c) feedback (explanations, validation, requests, etc.). Users send/receive information through the UI which translates user actions to a model-understandable format and vice versa. Our approach aims to specify this communications using interaction primitives.
• Figure 4: Definitions and visual representation for interaction primitives, types and actions. An action is defined as a primitive specified by a set of operations. Actions are communicated through messages, by specifying the interacting agents and the modifiers of the message. Actions are reusable and can be used by different messages. Sequences of messages can form patterns of interaction.
• Figure 5: Description of the HAI interactions using definitions of interaction primitives and patterns. The diagram illustrates our unpacking approach and the process of describing interaction primitives and patterns in terms of the interaction requirements.