Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains

TL;DR

This work addresses real-world robustness gaps by introducing ES-Studio and ImageNet-ES to study covariate shifts from environment and camera sensor factors during image acquisition. It demonstrates that current OOD detection methods struggle with covariate-shifted data and that learning perturbations in the environment and sensor domains can enhance robustness beyond conventional digital augmentations; moreover, sensor parameter control can significantly improve accuracy, sometimes rivaling larger models. The study proposes a Model-Specific OOD framework for covariate shifts and shows that sensor control can be a practical, model-agnostic way to mitigate distribution shifts without increasing model size. Collectively, ImageNet-ES provides a realistic benchmark for auditing OOD, domain generalization, and sensor-control strategies in computer vision, with implications for camera-enabled applications and future robustness research.

Abstract

Computer vision applications predict on digital images acquired by a camera from physical scenes through light. However, conventional robustness benchmarks rely on perturbations in digitized images, diverging from distribution shifts occurring in the image acquisition process. To bridge this gap, we introduce a new distribution shift dataset, ImageNet-ES, comprising variations in environmental and camera sensor factors by directly capturing 202k images with a real camera in a controllable testbed. With the new dataset, we evaluate out-of-distribution (OOD) detection and model robustness. We find that existing OOD detection methods do not cope with the covariate shifts in ImageNet-ES, implying that the definition and detection of OOD should be revisited to embrace real-world distribution shifts. We also observe that the model becomes more robust in both ImageNet-C and -ES by learning environment and sensor variations in addition to existing digital augmentations. Lastly, our results suggest that effective shift mitigation via camera sensor control can significantly improve performance without increasing model size. With these findings, our benchmark may aid future research on robustness, OOD, and camera sensor control for computer vision. Our code and dataset are available at https://github.com/Edw2n/ImageNet-ES.

Figures (8)

• Figure 1: Motivation and contribution of ImageNet-ES, the first benchmark on the necessary but unexplored faces of image covariate shifts: environment and camera sensor domains.
• Figure 2: Representative Examples of ImageNet-ES. In contrast to conventional robustness benchmarks that rely on digital perturbations, we directly capture 202k images by using a real camera in a controllable testbed. The dataset presents a wide range of covariate shifts caused by variations in light and camera sensor factors.
• Figure 3: Robustness improvement scenario to cover real-world C-OOD
• Figure 4: Illustration of the ES-Studio setup
• Figure 5: OOD score distribution with semantics-focused and MS-OOD frameworks. Tiny-ImageNet wu2017tiny and Texture Cimpoi2014CVPR are used for the ID and S-OOD datasets, respectively. ImageNet-ES serves as a C-OOD dataset.