Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Design Trajectory Map of Human-AI Collaborative Reinforcement Learning Systems: Survey and Taxonomy

Zhaoxing Li

TL;DR

The paper addresses the fragmented state of Collaborative Reinforcement Learning by offering a unified design framework and taxonomy. It introduces the CRL Design Trajectory Map to guide system design from high-level patterns to concrete algorithms, and defines five axes (Design Patterns, Levels/Parties, Capabilities, Interactive Methods, Algorithmic Models) to structure the literature. By synthesizing classic HCI design patterns with CRL methods, it provides concrete guidance for selecting architectures, interaction modalities, and learning paradigms, while outlining challenges and future directions such as safety, explainability, and data collection. The work aims to empower researchers and practitioners to design more efficient, trustworthy, and scalable human-AI collaborative systems across macro-to-micro levels.

Abstract

Driven by the algorithmic advancements in reinforcement learning and the increasing number of implementations of human-AI collaboration, Collaborative Reinforcement Learning (CRL) has been receiving growing attention. Despite this recent upsurge, this area is still rarely systematically studied. In this paper, we provide an extensive survey, investigating CRL methods based on both interactive reinforcement learning algorithms and human-AI collaborative frameworks that were proposed in the past decade. We elucidate and discuss via synergistic analysis methods both the growth of the field and the state-of-the-art; we conceptualise the existing frameworks from the perspectives of design patterns, collaborative levels, parties and capabilities, and review interactive methods and algorithmic models. Specifically, we create a new Human-AI CRL Design Trajectory Map, as a systematic modelling tool for the selection of existing CRL frameworks, as well as a method of designing new CRL systems, and finally of improving future CRL designs. Furthermore, we elaborate generic Human-AI CRL challenges, providing the research community with a guide towards novel research directions. The aim of this paper is to empower researchers with a systematic framework for the design of efficient and 'natural' human-AI collaborative methods, making it possible to work on maximised realisation of humans' and AI's potentials.

A Design Trajectory Map of Human-AI Collaborative Reinforcement Learning Systems: Survey and Taxonomy

TL;DR

The paper addresses the fragmented state of Collaborative Reinforcement Learning by offering a unified design framework and taxonomy. It introduces the CRL Design Trajectory Map to guide system design from high-level patterns to concrete algorithms, and defines five axes (Design Patterns, Levels/Parties, Capabilities, Interactive Methods, Algorithmic Models) to structure the literature. By synthesizing classic HCI design patterns with CRL methods, it provides concrete guidance for selecting architectures, interaction modalities, and learning paradigms, while outlining challenges and future directions such as safety, explainability, and data collection. The work aims to empower researchers and practitioners to design more efficient, trustworthy, and scalable human-AI collaborative systems across macro-to-micro levels.

Abstract

Driven by the algorithmic advancements in reinforcement learning and the increasing number of implementations of human-AI collaboration, Collaborative Reinforcement Learning (CRL) has been receiving growing attention. Despite this recent upsurge, this area is still rarely systematically studied. In this paper, we provide an extensive survey, investigating CRL methods based on both interactive reinforcement learning algorithms and human-AI collaborative frameworks that were proposed in the past decade. We elucidate and discuss via synergistic analysis methods both the growth of the field and the state-of-the-art; we conceptualise the existing frameworks from the perspectives of design patterns, collaborative levels, parties and capabilities, and review interactive methods and algorithmic models. Specifically, we create a new Human-AI CRL Design Trajectory Map, as a systematic modelling tool for the selection of existing CRL frameworks, as well as a method of designing new CRL systems, and finally of improving future CRL designs. Furthermore, we elaborate generic Human-AI CRL challenges, providing the research community with a guide towards novel research directions. The aim of this paper is to empower researchers with a systematic framework for the design of efficient and 'natural' human-AI collaborative methods, making it possible to work on maximised realisation of humans' and AI's potentials.
Paper Structure (54 sections, 3 equations, 5 figures, 1 table)

This paper contains 54 sections, 3 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology and Scope
  4. Literature Collection
  5. Human-AI Collaborative Reinforcement Learning Classification
  6. Human-AI Collaborative Reinforcement Learning Taxonomy
  7. Human-AI Collaborative Design Patterns
  8. CSE Pattern
  9. Bosch and Bronkhorst's Pattern
  10. Coactive Design Pattern
  11. Schmidt's Pattern
  12. Collaborative Levels and Parties
  13. Augmentative Level collaboration
  14. Human->AI
  15. AI->Human
  16. ...and 39 more sections

Figures (5)

  • Figure 1: Human-AI collaboration Design Model: From a human perspective, we focus on how humans interact with AI agents; from an AI agent's perspective, we focus on how AI agents accept human instructions or suggestions in algorithm implementation; and from a collaboration pattern perspective, we focus on what kind of way that humans and AI collaborate.
  • Figure 2: A new CRL taxonomy for interactive methods and design patterns
  • Figure 3: Triangle of different collaborative levels: the first level is Augmentative Level collaboration; the second level is Integrative Level collaboration; and the third level is Debative Level collaboration.
  • Figure 4: Different parties in the process of collaboration.
  • Figure 5: A Design Trajectory Map of Collaborative Reinforcement Learning Systems