Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery

Sara Al-Emadi, Yin Yang, Ferda Ofli

TL;DR

This work addresses the challenge of object detection under real-world distribution shifts in satellite imagery by proposing Real-World Distribution Shifts (RWDS), a DG benchmark suite with three datasets: RWDS-CZ (climate zones), RWDS-FR (flooded regions), and RWDS-HE (hurricane events). It conducts extensive evaluation of state-of-the-art detectors under single-source and multi-source training, highlighting that multi-source DG generally improves OOD generalisation and that Grounding DINO and GLIP often achieve the best trade-offs between ID and OOD performance. The study also provides a detailed error analysis and emphasizes the importance of robust DG benchmarks tailored to humanitarian and climate-change applications. RWDS and its accompanying codebase offer a valuable resource for benchmarking and improving the real-world robustness of satellite-imagery object detectors.

Abstract

Object detectors have achieved remarkable performance in many applications; however, these deep learning models are typically designed under the i.i.d. assumption, meaning they are trained and evaluated on data sampled from the same (source) distribution. In real-world deployment, however, target distributions often differ from source data, leading to substantial performance degradation. Domain Generalisation (DG) seeks to bridge this gap by enabling models to generalise to Out-Of-Distribution (OOD) data without access to target distributions during training, enhancing robustness to unseen conditions. In this work, we examine the generalisability and robustness of state-of-the-art object detectors under real-world distribution shifts, focusing particularly on spatial domain shifts. Despite the need, a standardised benchmark dataset specifically designed for assessing object detection under realistic DG scenarios is currently lacking. To address this, we introduce Real-World Distribution Shifts (RWDS), a suite of three novel DG benchmarking datasets that focus on humanitarian and climate change applications. These datasets enable the investigation of domain shifts across (i) climate zones and (ii) various disasters and geographic regions. To our knowledge, these are the first DG benchmarking datasets tailored for object detection in real-world, high-impact contexts. We aim for these datasets to serve as valuable resources for evaluating the robustness and generalisation of future object detection models. Our datasets and code are available at https://github.com/RWGAI/RWDS.

Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery

TL;DR

This work addresses the challenge of object detection under real-world distribution shifts in satellite imagery by proposing Real-World Distribution Shifts (RWDS), a DG benchmark suite with three datasets: RWDS-CZ (climate zones), RWDS-FR (flooded regions), and RWDS-HE (hurricane events). It conducts extensive evaluation of state-of-the-art detectors under single-source and multi-source training, highlighting that multi-source DG generally improves OOD generalisation and that Grounding DINO and GLIP often achieve the best trade-offs between ID and OOD performance. The study also provides a detailed error analysis and emphasizes the importance of robust DG benchmarks tailored to humanitarian and climate-change applications. RWDS and its accompanying codebase offer a valuable resource for benchmarking and improving the real-world robustness of satellite-imagery object detectors.

Abstract

Object detectors have achieved remarkable performance in many applications; however, these deep learning models are typically designed under the i.i.d. assumption, meaning they are trained and evaluated on data sampled from the same (source) distribution. In real-world deployment, however, target distributions often differ from source data, leading to substantial performance degradation. Domain Generalisation (DG) seeks to bridge this gap by enabling models to generalise to Out-Of-Distribution (OOD) data without access to target distributions during training, enhancing robustness to unseen conditions. In this work, we examine the generalisability and robustness of state-of-the-art object detectors under real-world distribution shifts, focusing particularly on spatial domain shifts. Despite the need, a standardised benchmark dataset specifically designed for assessing object detection under realistic DG scenarios is currently lacking. To address this, we introduce Real-World Distribution Shifts (RWDS), a suite of three novel DG benchmarking datasets that focus on humanitarian and climate change applications. These datasets enable the investigation of domain shifts across (i) climate zones and (ii) various disasters and geographic regions. To our knowledge, these are the first DG benchmarking datasets tailored for object detection in real-world, high-impact contexts. We aim for these datasets to serve as valuable resources for evaluating the robustness and generalisation of future object detection models. Our datasets and code are available at https://github.com/RWGAI/RWDS.

Paper Structure

This paper contains 40 sections, 5 equations, 9 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Object Detection
  4. Robustness Benchmarks
  5. Our RWDS Datasets
  6. RWDS Across Climate Zones
  7. RWDS in Disaster Damage Assessment
  8. RWDS Across Flooded Regions (RWDS-FR)
  9. RWDS Across Hurricane Events (RWDS-HE)
  10. Experiments
  11. Single-Source and Multi-Source Setup
  12. DG Evaluation Metrics
  13. Object Detectors and Hyperparameters
  14. Results and Analyses
  15. RWDS across Climate Zones (RWDS-CZ)
  16. ...and 25 more sections

Figures (9)

  • Figure 1: Example images from different climate zones
  • Figure 2: Class-wise distribution of training data for each domain as well as the overall data distribution across domains in RWDS-CZ
  • Figure 3: Embedding space representations of the RWDS datasets
  • Figure 4: Comparison of flood scenes between the US and India
  • Figure 5: Comparison of hurricane scenes from different events
  • ...and 4 more figures