AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems

TL;DR

This work organizes human–AI collaboration into Hybrid Decision Making Systems (HS) and a threefold taxonomy: Human Oversight, Learning to Abstain, and Learning Together. It formalizes HS with two agents $M$ and $H$, defines deferral and rejection policies, and contrasts staged versus joint learning architectures across L2R, L2D, and teacher–student paradigms. The survey catalogs a broad literature landscape, highlighting explainable AI, error-aware abstention, adaptive deferral to multiple experts, and bidirectional learning, while discussing practical costs, fairness, and validation challenges. The framework aims to guide the design of trustworthy, efficient, and fair hybrid systems with explicit attention to human costs, interaction artifacts, and the potential of LLMs as components of Learning Together approaches.

Abstract

Everyday we increasingly rely on machine learning models to automate and support high-stake tasks and decisions. This growing presence means that humans are now constantly interacting with machine learning-based systems, training and using models everyday. Several different techniques in computer science literature account for the human interaction with machine learning systems, but their classification is sparse and the goals varied. This survey proposes a taxonomy of Hybrid Decision Making Systems, providing both a conceptual and technical framework for understanding how current computer science literature models interaction between humans and machines.

Figures (5)

• Figure 1: Paradigms of hybrid systems, where human (circle) and machine (rectangle) steps alternate to form a cohesive system. In human overseers (\ref{['fig:paradigms:overseers']}), the machine performs a prediction, and the human accepts it or rejects it in favor of their own. In Learn to Abstain (\ref{['fig:paradigms:learn_to_abstain']}), an orchestrator assigns the prediction task to either of the two, which makes the prediction on their own. In Learning Together (\ref{['fig:paradigms:learn_together']}), the two agents engage in continuous interaction: the machine communicates its reasoning to the human, enabling the latter to understand the machine's internal mechanisms along with possibly rectifying any errors. The feedback loop continues until a human chooses to cease inspecting or interacting with the machine.
• Figure 2: Overview of the joint learning architecture for the Single-Expert Learning to Defer (L2D-SE) setting, illustrated in the application of flagging online contents for moderation. Adapted from Mozannar2023.
• Figure 3: An example of Question Answering machine with a hard reasoning language. The agent maps the question to a program using a set of primitives , and then executes the program on the input , providing the human with both a prediction, and a malleable program that they can correct.
• Figure 4: An example of a Question Answering hybrid system with a soft reasoning language. The human can interact with the machine through Natural Language. First, the machine provides a prediction and a rationale to the user, who then is able to correct the rationale and feed it back to the machine, updating its artifact bank ("Feedback memory" in DBLP:conf/emnlp/MishraTC22). On a subsequent interaction, the machine provides the correct prediction and rationale to the same question by leveraging the previous feedback.
• Figure 5: An example of a Question Answering hybrid system employing a soft reasoning language and leveraging Knowledge Graphs. Here, the machine consults its artifact bank to retrieve a supporting fact for its prediction, and provides it to the human agent alongside its prediction.