Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monitoring Human Dependence On AI Systems With Reliance Drills

Rosco Hunter, Richard Moulange, Jamie Bernardi, Merlin Stein

TL;DR

This paper addresses the risk that humans over-rely on AI in critical decisions and proposes reliance drills as a practical diagnostic and risk-management tool. It formally defines over-reliance and the drill mechanism, and offers a five-step pipeline for designing, implementing, and evaluating drills while balancing realism and safety. The authors discuss incentives for adoption, potential service models, and provide a medical setting exemplar to illustrate implementation and mitigations. The work aims to establish reliance drills as a standard practice to keep humans in the loop and mitigate AI-induced harm across domains.

Abstract

AI systems are assisting humans with increasingly diverse intellectual tasks but are still prone to mistakes. Humans are over-reliant on this assistance if they trust AI-generated advice, even though they would make a better decision on their own. To identify such instances of over-reliance, this paper proposes the reliance drill: an exercise that tests whether a human can recognise mistakes in AI-generated advice. Our paper examines the reasons why an organisation might choose to implement reliance drills and the doubts they may have about doing so. As an example, we consider the benefits and risks that could arise when using these drills to detect over-reliance on AI in healthcare professionals. We conclude by arguing that reliance drills should become a standard risk management practice for ensuring humans remain appropriately involved in the oversight of AI-assisted decisions.

Monitoring Human Dependence On AI Systems With Reliance Drills

TL;DR

This paper addresses the risk that humans over-rely on AI in critical decisions and proposes reliance drills as a practical diagnostic and risk-management tool. It formally defines over-reliance and the drill mechanism, and offers a five-step pipeline for designing, implementing, and evaluating drills while balancing realism and safety. The authors discuss incentives for adoption, potential service models, and provide a medical setting exemplar to illustrate implementation and mitigations. The work aims to establish reliance drills as a standard practice to keep humans in the loop and mitigate AI-induced harm across domains.

Abstract

AI systems are assisting humans with increasingly diverse intellectual tasks but are still prone to mistakes. Humans are over-reliant on this assistance if they trust AI-generated advice, even though they would make a better decision on their own. To identify such instances of over-reliance, this paper proposes the reliance drill: an exercise that tests whether a human can recognise mistakes in AI-generated advice. Our paper examines the reasons why an organisation might choose to implement reliance drills and the doubts they may have about doing so. As an example, we consider the benefits and risks that could arise when using these drills to detect over-reliance on AI in healthcare professionals. We conclude by arguing that reliance drills should become a standard risk management practice for ensuring humans remain appropriately involved in the oversight of AI-assisted decisions.
Paper Structure (5 sections, 3 figures, 3 tables)

This paper contains 5 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Defining reliance drills
  3. Incentives for reliance drills
  4. A pipeline for reliance drills
  5. Applying reliance drills in a medical setting

Figures (3)

  • Figure 1: During a reliance drill, an investigator forces an AI system to underperform---typically by prompting it to purposefully make a mistake---and then uses this to identify over-reliant users.
  • Figure 2: Forethought and risk assessments are needed for safe and effective reliance drills. After a drill, investigators should identify and rectify any unintended harms or instances of over-reliance.
  • Figure 3: There are appropriate, benign, and undesirable outcomes of a human-AI interaction. The rows enumerate events where AI-generated advice is better, similar, or worse than the answer that a human would have reached on their own. The columns represent the human's response to this advice.