Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study

Luca Giamattei, Matteo Biagiola, Roberto Pietrantuono, Stefano Russo, Paolo Tonella

TL;DR

This replication and extension of an empirical study of Reinforcement Learning shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured.

Abstract

In a recent study, Reinforcement Learning (RL) used in combination with many-objective search, has been shown to outperform alternative techniques (random search and many-objective search) for online testing of Deep Neural Network-enabled systems. The empirical evaluation of these techniques was conducted on a state-of-the-art Autonomous Driving System (ADS). This work is a replication and extension of that empirical study. Our replication shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured. Our extension aims at eliminating some of the possible reasons for the poor performance of RL observed in our replication: (1) the presence of reward components providing contrasting or useless feedback to the RL agent; (2) the usage of an RL algorithm (Q-learning) which requires discretization of an intrinsically continuous state space. Results show that our new RL agent is able to converge to an effective policy that outperforms random testing. Results also highlight other possible improvements, which open to further investigations on how to best leverage RL for online ADS testing.

Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study

TL;DR

This replication and extension of an empirical study of Reinforcement Learning shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured.

Abstract

In a recent study, Reinforcement Learning (RL) used in combination with many-objective search, has been shown to outperform alternative techniques (random search and many-objective search) for online testing of Deep Neural Network-enabled systems. The empirical evaluation of these techniques was conducted on a state-of-the-art Autonomous Driving System (ADS). This work is a replication and extension of that empirical study. Our replication shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured. Our extension aims at eliminating some of the possible reasons for the poor performance of RL observed in our replication: (1) the presence of reward components providing contrasting or useless feedback to the RL agent; (2) the usage of an RL algorithm (Q-learning) which requires discretization of an intrinsically continuous state space. Results show that our new RL agent is able to converge to an effective policy that outperforms random testing. Results also highlight other possible improvements, which open to further investigations on how to best leverage RL for online ADS testing.
Paper Structure (21 sections, 1 equation, 7 figures, 8 tables)

This paper contains 21 sections, 1 equation, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning
  4. Replication study
  5. Problem definition
  6. Subject and configuration
  7. Replication
  8. Methodology
  9. Effectiveness
  10. Efficiency
  11. Discussion
  12. Extension study
  13. Problem definition
  14. Experiments
  15. Evaluation criteria
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Coverage of safety and functional requirements
  • Figure 2: Average coverage of safety and functional requirements over time.
  • Figure 3: Average Q-Table dimension (number of observed states) over number of steps.
  • Figure 4: Box plot of the distribution of the number of violations of requirement R2 (collisions with Vehicle in Front).
  • Figure 5: Average number of violations (i.e., collision with vehicles) over time.
  • ...and 2 more figures