Classification Drives Geographic Bias in Street Scene Segmentation

TL;DR

This work probes geo-bias in street-scene instance segmentation by testing Eurocentric Cityscapes-trained models on Mapillary Vistas across six continents. It introduces continent-IoU and a class-merging approach to quantify and isolate the influence of classification errors on geo-bias, yielding a geo-disparity measure $Disp = $ $\sigma_{continents} / \mu_{continents}$. The findings show that classification errors account for substantial portions of geo-bias in detection ($19$-$88\%$) and segmentation ($10$-$90\%$), while localization biases remain comparatively small. Practically, the study suggests adopting coarser labels (e.g., four-wheeler) to improve global applicability of region-specific models, while highlighting the need for further work to address localization biases and automate class-merging.

Abstract

Previous studies showed that image datasets lacking geographic diversity can lead to biased performance in models trained on them. While earlier work studied general-purpose image datasets (e.g., ImageNet) and simple tasks like image recognition, we investigated geo-biases in real-world driving datasets on a more complex task: instance segmentation. We examined if instance segmentation models trained on European driving scenes (Eurocentric models) are geo-biased. Consistent with previous work, we found that Eurocentric models were geo-biased. Interestingly, we found that geo-biases came from classification errors rather than localization errors, with classification errors alone contributing 10-90% of the geo-biases in segmentation and 19-88% of the geo-biases in detection. This showed that while classification is geo-biased, localization (including detection and segmentation) is geographically robust. Our findings show that in region-specific models (e.g., Eurocentric models), geo-biases from classification errors can be significantly mitigated by using coarser classes (e.g., grouping car, bus, and truck as 4-wheeler).

Figures (5)

• Figure 1: Evaluating Eurocentric models on the Mapillary Vistas dataset
• Figure 2: Detection and segmentation box plots. Each box plot is made with $6$ data points. Here, a data point refers to the average performance (IoU) of a model for a class in a continent. In both figures, while person and car hold minimal geo-biases, other classes (rider, motorcycle, bicycle, bus, and truck) have significant geo-biases.
• Figure 3: Detection and segmentation box plots after class-merging. Class-merging was applied to the following groups: car-bus-truck, motorcycle-bicycle, and person-rider.
• Figure 4: Misclassifications by a Eurocentric model (ViT-Swin-L) on images from Africa (left two) and Asia (right one). Objects circled in red were misclassified.
• Figure 5: An example figure with a bus icon to demonstrate how class-merging resolves misclassification of the bus class and calculates a corrected-bus-IoU.