Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations

Ahmed Hammam, Bharathwaj Krishnaswami Sreedhar, Nura Kawa, Tim Patzelt, Oliver De Candido

TL;DR

By incorporating augmentations, this work observes enhanced robustness of ML-based perception models, making them more resilient to edge cases encountered in real-world ODDs, underlines the importance of customized augmentations and offers an effective solution for improving the safety and reliability of autonomous driving functions.

Abstract

Advancing Machine Learning (ML)-based perception models for autonomous systems necessitates addressing weak spots within the models, particularly in challenging Operational Design Domains (ODDs). These are environmental operating conditions of an autonomous vehicle which can contain difficult conditions, e.g., lens flare at night or objects reflected in a wet street. This report introduces a novel methodology for training with augmentations to enhance model robustness and performance in such conditions. The proposed approach leverages customized physics-based augmentation functions, to generate realistic training data that simulates diverse ODD scenarios. We present a comprehensive framework that includes identifying weak spots in ML models, selecting suitable augmentations, and devising effective training strategies. The methodology integrates hyperparameter optimization and latent space optimization to fine-tune augmentation parameters, ensuring they maximally improve the ML models' performance. Experimental results demonstrate improvements in model performance, as measured by commonly used metrics such as mean Average Precision (mAP) and mean Intersection over Union (mIoU) on open-source object detection and semantic segmentation models and datasets. Our findings emphasize that optimal training strategies are model- and data-specific and highlight the benefits of integrating augmentations into the training pipeline. By incorporating augmentations, we observe enhanced robustness of ML-based perception models, making them more resilient to edge cases encountered in real-world ODDs. This work underlines the importance of customized augmentations and offers an effective solution for improving the safety and reliability of autonomous driving functions.

Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations

TL;DR

By incorporating augmentations, this work observes enhanced robustness of ML-based perception models, making them more resilient to edge cases encountered in real-world ODDs, underlines the importance of customized augmentations and offers an effective solution for improving the safety and reliability of autonomous driving functions.

Abstract

Advancing Machine Learning (ML)-based perception models for autonomous systems necessitates addressing weak spots within the models, particularly in challenging Operational Design Domains (ODDs). These are environmental operating conditions of an autonomous vehicle which can contain difficult conditions, e.g., lens flare at night or objects reflected in a wet street. This report introduces a novel methodology for training with augmentations to enhance model robustness and performance in such conditions. The proposed approach leverages customized physics-based augmentation functions, to generate realistic training data that simulates diverse ODD scenarios. We present a comprehensive framework that includes identifying weak spots in ML models, selecting suitable augmentations, and devising effective training strategies. The methodology integrates hyperparameter optimization and latent space optimization to fine-tune augmentation parameters, ensuring they maximally improve the ML models' performance. Experimental results demonstrate improvements in model performance, as measured by commonly used metrics such as mean Average Precision (mAP) and mean Intersection over Union (mIoU) on open-source object detection and semantic segmentation models and datasets. Our findings emphasize that optimal training strategies are model- and data-specific and highlight the benefits of integrating augmentations into the training pipeline. By incorporating augmentations, we observe enhanced robustness of ML-based perception models, making them more resilient to edge cases encountered in real-world ODDs. This work underlines the importance of customized augmentations and offers an effective solution for improving the safety and reliability of autonomous driving functions.
Paper Structure (40 sections, 4 equations, 6 figures, 3 tables)

This paper contains 40 sections, 4 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Data Augmentation and Synthetic Data Generation
  4. Fine-Tuning Pretrained Models
  5. Curriculum Learning and Self-Paced Learning
  6. Layer-Wise Fine-Tuning
  7. Learning Rate Scheduling in Fine-Tuning Pretrained Models
  8. Regularization Techniques in Fine-Tuning Pretrained Models
  9. Hybrid Approaches in Fine-Tuning Pretrained Models
  10. Methodology – Augmented Training Strategy
  11. Overview of the Augmented Training Strategy
  12. Weak Spot Identification
  13. Augmentation Selection and Fine-Tuning
  14. Augmented Training Process
  15. Robustness Evaluation
  16. ...and 25 more sections

Figures (6)

  • Figure 1: General flowchart for designing an experiment from lawson2014design.
  • Figure 2: Visualization of the template for designing an augmented training experiment.
  • Figure 3: Example images from dataset zod_dataset. (a), (b) are images collected during rain, while (c), (d) were collected during clear weather conditions.
  • Figure 4: Examples of clear weather images from the Cityscapes dataset cityscapes_dataset.
  • Figure 5: Examples of rain images from the dataset acdc_dataset.
  • ...and 1 more figures