From Snow to Rain: Evaluating Robustness, Calibration, and Complexity of Model-Based Robust Training

TL;DR

This work addresses the challenge of robustness to natural corruptions in safety-critical perception by evaluating a family of model-based nuisance-variation training methods and their hybrids against traditional analytical augmentations. Using the CURE-TSR dataset with Snow and Rain corruptions, it compares three model-based approaches—MDA, MRT, and MAT—and two hybrids, across accuracy, calibration, and training complexity. The findings show consistent robustness and calibration gains from model-based methods, with MAT delivering the strongest overall performance at higher computational cost, and MDA offering a favorable efficiency–robustness trade-off; training severity further modulates outcomes, with severity 2 favoring calibration and mild corruption, and severity 5 enhancing extreme-corruption robustness. The results underscore the value of learned nuisance models for capturing natural variability and suggest a promising path toward more resilient perception systems, while outlining practical guidelines for balancing performance and cost in real-world training pipelines.

Abstract

Robustness to natural corruptions remains a critical challenge for reliable deep learning, particularly in safety-sensitive domains. We study a family of model-based training approaches that leverage a learned nuisance variation model to generate realistic corruptions, as well as new hybrid strategies that combine random coverage with adversarial refinement in nuisance space. Using the Challenging Unreal and Real Environments for Traffic Sign Recognition dataset (CURE-TSR), with Snow and Rain corruptions, we evaluate accuracy, calibration, and training complexity across corruption severities. Our results show that model-based methods consistently outperform baselines Vanilla, Adversarial Training, and AugMix baselines, with model-based adversarial training providing the strongest robustness under across all corruptions but at the expense of higher computation and model-based data augmentation achieving comparable robustness with $T$ less computational complexity without incurring a statistically significant drop in performance. These findings highlight the importance of learned nuisance models for capturing natural variability, and suggest a promising path toward more resilient and calibrated models under challenging conditions.

Figures (5)

• Figure 1: CURE-TSR under Snow corruption. Vanilla and AT degrade quickly with severity, while AugMix collapses even more sharply. Model-based methods (MDA, MRT, MAT) sustain substantially higher robustness and lower ECE, retaining $\sim 75\%$ accuracy and $\sim 20\%$ ECE at severity 5 compared to $\sim 55$--$65\%$ and $30$--$32\%$ for baselines.
• Figure 2: CURE-TSR under Rain corruption. Rain is more challenging than Snow, causing Vanilla, AT, and AugMix to collapse below $60\%$ accuracy and above $30\%$ ECE at severity 5. Model-based methods (MDA, MRT, MAT) markedly improve robustness: MRT5 and MDA5 reach $\sim 59\%$, while MAT5 achieves $62.8\%$ with the best calibration ($20.6\%$ ECE).
• Figure 3: Comparing effect of Snow corruption training severity. We compare variants of MDA, MRT, and MAT trained with severity 2 and severity 5 augmentations. Results show that severity 5 variants consistently achieve higher robustness at moderate to high corruption levels, while severity 2 variants perform slightly better under milder corruptions for Snow corruption.
• Figure 4: Comparing effect of Rain corruption training severity. We compare variants of MDA, MRT, and MAT trained with severity 2 and severity 5 augmentations. Results show that severity 5 variants consistently achieve higher robustness at moderate to high corruption levels, while severity 2 variants perform slightly better under milder corruptions for Snow corruption. Interestingly MAT2 under Rain, outperformed all the methods trained with severity 5.
• Figure 5: Empirical training times (in minutes) for baseline methods. Vanilla ERM is fastest, followed by AT and MDA. MRT and MAT are substantially more expensive due to repeated nuisance sampling and adversarial optimization.