Genetic Learning for Designing Sim-to-Real Data Augmentations

TL;DR

Data augmentations are used to close the sim-to-real gap in object detection by widening the synthetic training distribution. The authors introduce two feature-space metrics—a per-feature variance measure and a Wasserstein-1 distance between synthetic and real feature distributions—and show they strongly predict real-data performance when used to guide augmentation design. They then present GeneticAugment, a genetic-programming method that automatically evolves augmentation policies to maximize these metrics without training models, achieving strong sim-to-real gains on Sim10k→Cityscapes and outperforming several non-data-driven baselines and many domain-adaptive techniques. The work offers a scalable, model-agnostic approach with released code for reproducibility.

Abstract

Data augmentations are useful in closing the sim-to-real domain gap when training on synthetic data. This is because they widen the training data distribution, thus encouraging the model to generalize better to other domains. Many image augmentation techniques exist, parametrized by different settings, such as strength and probability. This leads to a large space of different possible augmentation policies. Some policies work better than others for overcoming the sim-to-real gap for specific datasets, and it is unclear why. This paper presents two different interpretable metrics that can be combined to predict how well a certain augmentation policy will work for a specific sim-to-real setting, focusing on object detection. We validate our metrics by training many models with different augmentation policies and showing a strong correlation with performance on real data. Additionally, we introduce GeneticAugment, a genetic programming method that can leverage these metrics to automatically design an augmentation policy for a specific dataset without needing to train a model.

Figures (7)

• Figure 1: Performance of different augmentations on Sim10k $\rightarrow$ Cityscapes.
• Figure 2: Performance combinations of augmentations on Sim10k $\rightarrow$ Cityscapes.
• Figure 3: Relationship between the proposed metrics measured for augmentations and performance of models trained with those augmentations on Sim10k $\rightarrow$ Cityscapes.
• Figure 4: Evolution of the range of the metrics measured on the population during the generations of the genetic algorithm.
• Figure 5: Relationship between the metrics proposed by yamaguchi2019effectiveda and performance of models trained with different augmentations on Sim10k $\rightarrow$ Cityscapes.