Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators

Dmytro Humeniuk, Houssem Ben Braiek, Thomas Reid, Foutse Khomh

TL;DR

The MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework is proposed, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws.

Abstract

Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. The creation and assessment of this model requires real world data, which can be hard to label and collect, especially when the hardware setup is not available. The current techniques primarily focus on using synthetic data to train deep neural networks (DDNs) and identifying failures through offline or online simulation-based testing. However, the process of exploiting the identified failures to uncover design flaws early on, and leveraging the optimized DNN within the simulation to accelerate the engineering of the DNN for real-world tasks remains unclear. To address these challenges, we propose the MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws. Evaluation of two industrial case studies demonstrated that MARTENS effectively reveals robotic manipulator system failures, detecting 25 % to 50 % more failures with greater diversity compared to random test generation. The model trained and repaired using the MARTENS approach achieved mean average precision (mAP) scores of 0.91 and 0.82 on real-world images with no prior retraining. Further fine-tuning on real-world images for a few epochs (less than 10) increased the mAP to 0.95 and 0.89 for the first and second use cases, respectively. In contrast, a model trained solely on real-world data achieved mAPs of 0.8 and 0.75 for use case 1 and use case 2 after more than 25 epochs.

In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators

TL;DR

The MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework is proposed, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws.

Abstract

Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. The creation and assessment of this model requires real world data, which can be hard to label and collect, especially when the hardware setup is not available. The current techniques primarily focus on using synthetic data to train deep neural networks (DDNs) and identifying failures through offline or online simulation-based testing. However, the process of exploiting the identified failures to uncover design flaws early on, and leveraging the optimized DNN within the simulation to accelerate the engineering of the DNN for real-world tasks remains unclear. To address these challenges, we propose the MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws. Evaluation of two industrial case studies demonstrated that MARTENS effectively reveals robotic manipulator system failures, detecting 25 % to 50 % more failures with greater diversity compared to random test generation. The model trained and repaired using the MARTENS approach achieved mean average precision (mAP) scores of 0.91 and 0.82 on real-world images with no prior retraining. Further fine-tuning on real-world images for a few epochs (less than 10) increased the mAP to 0.95 and 0.89 for the first and second use cases, respectively. In contrast, a model trained solely on real-world data achieved mAPs of 0.8 and 0.75 for use case 1 and use case 2 after more than 25 epochs.

Paper Structure

This paper contains 17 sections, 2 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. In-simulation Test Environment Specification
  4. Search-based Test Generation
  5. Test Evaluation
  6. Vision Model with Randomized Synthetic Data
  7. Model Repair Using Failure Data
  8. Evaluation
  9. Experimental Setup
  10. Research Questions
  11. RQ1: Effectiveness of MARTENS in revealing ARM failures
  12. RQ2: Usefulness of MARTENS in repairing DNN's inefficiencies
  13. RQ3: Sim-to-real ransferability of the in-simulation optimized DNN
  14. Related work
  15. Simulation-based Testing
  16. ...and 2 more sections

Figures (9)

  • Figure 1: The diagram of the MARTENS approach
  • Figure 2: Camera view in the virtual environment
  • Figure 3: Illustrations of the test subjects used in the study
  • Figure 4: Top views from the fixed camera for the subjects under test with model predictions
  • Figure 5: Number of failures revealed over time
  • ...and 4 more figures