Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to design a dataset compliant with an ML-based system ODD?

Cyril Cappi, Noémie Cohen, Mélanie Ducoffe, Christophe Gabreau, Laurent Gardes, Adrien Gauffriau, Jean-Brice Ginestet, Franck Mamalet, Vincent Mussot, Claire Pagetti, David Vigouroux

TL;DR

The replicable framework presented in this paper addresses the challenges of designing a dataset compliant with the stringent needs of ML-based systems certification in safety-critical applications.

Abstract

This paper focuses on a Vision-based Landing task and presents the design and the validation of a dataset that would comply with the Operational Design Domain (ODD) of a Machine-Learning (ML) system. Relying on emerging certification standards, we describe the process for establishing ODDs at both the system and image levels. In the process, we present the translation of high-level system constraints into actionable image-level properties, allowing for the definition of verifiable Data Quality Requirements (DQRs). To illustrate this approach, we use the Landing Approach Runway Detection (LARD) dataset which combines synthetic imagery and real footage, and we focus on the steps required to verify the DQRs. The replicable framework presented in this paper addresses the challenges of designing a dataset compliant with the stringent needs of ML-based systems certification in safety-critical applications.

How to design a dataset compliant with an ML-based system ODD?

TL;DR

The replicable framework presented in this paper addresses the challenges of designing a dataset compliant with the stringent needs of ML-based systems certification in safety-critical applications.

Abstract

This paper focuses on a Vision-based Landing task and presents the design and the validation of a dataset that would comply with the Operational Design Domain (ODD) of a Machine-Learning (ML) system. Relying on emerging certification standards, we describe the process for establishing ODDs at both the system and image levels. In the process, we present the translation of high-level system constraints into actionable image-level properties, allowing for the definition of verifiable Data Quality Requirements (DQRs). To illustrate this approach, we use the Landing Approach Runway Detection (LARD) dataset which combines synthetic imagery and real footage, and we focus on the steps required to verify the DQRs. The replicable framework presented in this paper addresses the challenges of designing a dataset compliant with the stringent needs of ML-based systems certification in safety-critical applications.
Paper Structure (24 sections, 13 figures, 1 table)

This paper contains 24 sections, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Certification guidelines
  3. Motivation
  4. Approach
  5. System-level ODD
  6. ODD concept
  7. ED-324/ARP6983 -- operating environment
  8. VBL system-level ODD
  9. VBL ML constituent
  10. Image-level ODD
  11. ED-324/ARP6983 -- MLCODD Characterization and validation
  12. Approach to design an image-level ODD
  13. Geometry parameters
  14. Domain-specific concepts
  15. DQR for VBL
  16. ...and 9 more sections

Figures (13)

  • Figure 1: Illustration of the quality of the synthetic images - Comparison of a real landing footage (left) with synthetic replicas (Google Earth Studio center, Microsoft Flight Simulator right)
  • Figure 2: ODD Design Workflow
  • Figure 3: Geometry of a landing
  • Figure 4: VBL constituent architecture with 3 stages
  • Figure 5: Generator pipeline
  • ...and 8 more figures

Theorems & Definitions (1)

  • Definition 1: Generic landing approach cone