Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated System-level Testing of Unmanned Aerial Systems

Hassan Sartaj, Asmar Muqeet, Muhammad Zohaib Iqbal, Muhammad Uzair Khan

TL;DR

The paper addresses the need for scalable automated system-level testing in UAS safety-critical avionics. It introduces AITester, which combines model-based testing with deep reinforcement learning to automatically generate, execute, and evaluate test scenarios at runtime using UML-based models and OCL constraints. The authors provide a modeling methodology with two UML profiles, a public toolset, and empirical evaluation on ArduCopter and GCS-CDS that demonstrates enhanced fault-revealing test generation and diverse testing paths. The work demonstrates the feasibility of automated system-level UAS testing and outlines future directions such as ensuring model correctness, learning state machines, handling complex constraints, and extending to UAV swarms.

Abstract

Unmanned aerial systems (UAS) rely on various avionics systems that are safety-critical and mission-critical. A major requirement of international safety standards is to perform rigorous system-level testing of avionics software systems. The current industrial practice is to manually create test scenarios, manually/automatically execute these scenarios using simulators, and manually evaluate outcomes. The test scenarios typically consist of setting certain flight or environment conditions and testing the system under test in these settings. The state-of-the-art approaches for this purpose also require manual test scenario development and evaluation. In this paper, we propose a novel approach to automate the system-level testing of the UAS. The proposed approach (AITester) utilizes model-based testing and artificial intelligence (AI) techniques to automatically generate, execute, and evaluate various test scenarios. The test scenarios are generated on the fly, i.e., during test execution based on the environmental context at runtime. The approach is supported by a toolset. We empirically evaluate the proposed approach on two core components of UAS, an autopilot system of an unmanned aerial vehicle (UAV) and cockpit display systems (CDS) of the ground control station (GCS). The results show that the AITester effectively generates test scenarios causing deviations from the expected behavior of the UAV autopilot and reveals potential flaws in the GCS-CDS.

Automated System-level Testing of Unmanned Aerial Systems

TL;DR

The paper addresses the need for scalable automated system-level testing in UAS safety-critical avionics. It introduces AITester, which combines model-based testing with deep reinforcement learning to automatically generate, execute, and evaluate test scenarios at runtime using UML-based models and OCL constraints. The authors provide a modeling methodology with two UML profiles, a public toolset, and empirical evaluation on ArduCopter and GCS-CDS that demonstrates enhanced fault-revealing test generation and diverse testing paths. The work demonstrates the feasibility of automated system-level UAS testing and outlines future directions such as ensuring model correctness, learning state machines, handling complex constraints, and extending to UAV swarms.

Abstract

Unmanned aerial systems (UAS) rely on various avionics systems that are safety-critical and mission-critical. A major requirement of international safety standards is to perform rigorous system-level testing of avionics software systems. The current industrial practice is to manually create test scenarios, manually/automatically execute these scenarios using simulators, and manually evaluate outcomes. The test scenarios typically consist of setting certain flight or environment conditions and testing the system under test in these settings. The state-of-the-art approaches for this purpose also require manual test scenario development and evaluation. In this paper, we propose a novel approach to automate the system-level testing of the UAS. The proposed approach (AITester) utilizes model-based testing and artificial intelligence (AI) techniques to automatically generate, execute, and evaluate various test scenarios. The test scenarios are generated on the fly, i.e., during test execution based on the environmental context at runtime. The approach is supported by a toolset. We empirically evaluate the proposed approach on two core components of UAS, an autopilot system of an unmanned aerial vehicle (UAV) and cockpit display systems (CDS) of the ground control station (GCS). The results show that the AITester effectively generates test scenarios causing deviations from the expected behavior of the UAV autopilot and reveals potential flaws in the GCS-CDS.
Paper Structure (61 sections, 5 equations, 15 figures, 6 tables)

This paper contains 61 sections, 5 equations, 15 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Unmanned Aerial Systems (UAS)
  4. Model-based Testing
  5. Reinforcement Learning (RL)
  6. Deep Learning
  7. Approach
  8. Overview of the Approach
  9. Modeling Methodology for UAS
  10. UAS Structural Profile
  11. UAV Behavioral Profile
  12. Modeling Guidelines
  13. Modeling of UAS SUT
  14. UAS SUT Domain Modeling
  15. Flight Behavior Modeling
  16. ...and 46 more sections

Figures (15)

  • Figure 1: An agent's interaction with the environment sutton2018reinforcement
  • Figure 2: An overview of the approach showing the proposed UAS profile, inputs required from avionics testers, AITester, and UAV Environment. The dotted arrows () represent the compliance of models to the UAS profile and solid arrows () represent the information/control flow.
  • Figure 3: UAS profile for modeling structural details
  • Figure 4: An excerpt of UAV behavioral profile for modeling flight states
  • Figure 5: An excerpt of UAV behavioral profile for modeling flight actions
  • ...and 10 more figures