Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What's my role? Modelling responsibility for AI-based safety-critical systems

Philippa Ryan, Zoe Porter, Joanna Al-Qaddoumi, John McDermid, Ibrahim Habli

TL;DR

AI-Safety-Critical Systems (AI-SCS) raise responsibility gaps across diverse actors and lifecycles. The paper offers a practical responsibility modelling approach that fuses Hart's responsibility senses with Lock and Porter et al.'s notation to map actors, occurrences, and resources across development and operation. It demonstrates the method via two case studies—the Tempe autonomous vehicle crash and a diabetes co-morbidity predictive tool—to reveal gaps, risk transfers, and liability considerations, informing safer deployment and just culture. The work provides a basis for Safety Assurance Cases (SACs) and supports proactive risk management, accountability, and learning for future AI-SCS developments.

Abstract

AI-Based Safety-Critical Systems (AI-SCS) are being increasingly deployed in the real world. These can pose a risk of harm to people and the environment. Reducing that risk is an overarching priority during development and operation. As more AI-SCS become autonomous, a layer of risk management via human intervention has been removed. Following an accident it will be important to identify causal contributions and the different responsible actors behind those to learn from mistakes and prevent similar future events. Many authors have commented on the "responsibility gap" where it is difficult for developers and manufacturers to be held responsible for harmful behaviour of an AI-SCS. This is due to the complex development cycle for AI, uncertainty in AI performance, and dynamic operating environment. A human operator can become a "liability sink" absorbing blame for the consequences of AI-SCS outputs they weren't responsible for creating, and may not have understanding of. This cross-disciplinary paper considers different senses of responsibility (role, moral, legal and causal), and how they apply in the context of AI-SCS safety. We use a core concept (Actor(A) is responsible for Occurrence(O)) to create role responsibility models, producing a practical method to capture responsibility relationships and provide clarity on the previously identified responsibility issues. Our paper demonstrates the approach with two examples: a retrospective analysis of the Tempe Arizona fatal collision involving an autonomous vehicle, and a safety focused predictive role-responsibility analysis for an AI-based diabetes co-morbidity predictor. In both examples our primary focus is on safety, aiming to reduce unfair or disproportionate blame being placed on operators or developers. We present a discussion and avenues for future research.

What's my role? Modelling responsibility for AI-based safety-critical systems

TL;DR

AI-Safety-Critical Systems (AI-SCS) raise responsibility gaps across diverse actors and lifecycles. The paper offers a practical responsibility modelling approach that fuses Hart's responsibility senses with Lock and Porter et al.'s notation to map actors, occurrences, and resources across development and operation. It demonstrates the method via two case studies—the Tempe autonomous vehicle crash and a diabetes co-morbidity predictive tool—to reveal gaps, risk transfers, and liability considerations, informing safer deployment and just culture. The work provides a basis for Safety Assurance Cases (SACs) and supports proactive risk management, accountability, and learning for future AI-SCS developments.

Abstract

AI-Based Safety-Critical Systems (AI-SCS) are being increasingly deployed in the real world. These can pose a risk of harm to people and the environment. Reducing that risk is an overarching priority during development and operation. As more AI-SCS become autonomous, a layer of risk management via human intervention has been removed. Following an accident it will be important to identify causal contributions and the different responsible actors behind those to learn from mistakes and prevent similar future events. Many authors have commented on the "responsibility gap" where it is difficult for developers and manufacturers to be held responsible for harmful behaviour of an AI-SCS. This is due to the complex development cycle for AI, uncertainty in AI performance, and dynamic operating environment. A human operator can become a "liability sink" absorbing blame for the consequences of AI-SCS outputs they weren't responsible for creating, and may not have understanding of. This cross-disciplinary paper considers different senses of responsibility (role, moral, legal and causal), and how they apply in the context of AI-SCS safety. We use a core concept (Actor(A) is responsible for Occurrence(O)) to create role responsibility models, producing a practical method to capture responsibility relationships and provide clarity on the previously identified responsibility issues. Our paper demonstrates the approach with two examples: a retrospective analysis of the Tempe Arizona fatal collision involving an autonomous vehicle, and a safety focused predictive role-responsibility analysis for an AI-based diabetes co-morbidity predictor. In both examples our primary focus is on safety, aiming to reduce unfair or disproportionate blame being placed on operators or developers. We present a discussion and avenues for future research.
Paper Structure (19 sections, 7 figures, 2 tables)

This paper contains 19 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and related literature
  3. AI-based Safety critical systems and responsibility gaps
  4. Responsibility complexity in AI-SCS development and operation
  5. Legal pitfalls and due diligence
  6. Clarifying types of responsibility for AI-SCS
  7. Summary
  8. Method
  9. Notation
  10. Creating and analysing the model
  11. Gathering information
  12. Reviewing and Analysing the model
  13. Example 1 - Fatal collision with an Autonomous Vehicle in Tempe, Arizona
  14. Producing and analysing the collision responsibility model
  15. Example 2 - Diabetes co-morbidity clinical decision support AI-SCS
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Responsibility model elements
  • Figure 2: Development of a simple responsibility model
  • Figure 3: Initial responsibility model showing actors and roles from Uber ATG crash in Tempe
  • Figure 4: Extended Uber Tempe responsibility model highlighting some responsibility findings from Uber ATG crash in Tempe uber
  • Figure 5: DCP explainability example showing influence of FOIs DCP_safecomp
  • ...and 2 more figures