Enhancing Worldwide Image Geolocation by Ensembling Satellite-Based Ground-Level Attribute Predictors

TL;DR

A novel metric, Recall vs Area (RvA), which assesses distributions of location estimates in image geolocation results in a manner similar to precision-recall in document retrieval, measuring recall as a function of area.

Abstract

We examine the challenge of estimating the location of a single ground-level image in the absence of GPS or other location metadata. Currently, geolocation systems are evaluated by measuring the Great Circle Distance between the predicted location and ground truth. Because this measurement only uses a single point, it cannot assess the distribution of predictions by geolocation systems. Evaluation of a distribution of potential locations (areas) is required when there are follow-on procedures to further narrow down or verify the location. This is especially important in poorly-sampled regions e.g. rural and wilderness areas. In this paper, we introduce a novel metric, Recall vs Area (RvA), which measures the accuracy of estimated distributions of locations. RvA treats image geolocation results similarly to document retrieval, measuring recall as a function of area: For a ranked list of (possibly discontiguous) predicted regions, we measure the area required for accumulated regions to contain the ground truth coordinate. This produces a curve similar to a precision-recall curve, where "precision" is replaced by square kilometers area, enabling evaluation for different downstream search area budgets. Following from this view of the problem, we then examine an ensembling approach to global-scale image geolocation, which incorporates information from multiple sources, and can readily incorporate multiple models, attribute predictors, and data sources. We study its effectiveness by combining the geolocation models GeoEstimation and the current state-of-the-art, GeoCLIP, with attribute predictors based on Oak Ridge National Laboratory LandScan and European Space Agency Climate Change Initiative Land Cover. We find significant improvements in image geolocation for areas that are under-represented in the training set, particularly non-urban areas, on both Im2GPS3k and Street View images.

Figures (6)

• Figure 1: (A) Visualization of 'low' population density mask obtained from ORNL LandScan global, see Sec. \ref{['sec:predictors']} for discussion. Masks obtained from LandScan as well as ESA-CCI Land Cover are used by our ensembling method to improve image location estimates of base geolocation models. (B) The objective of our proposed ensembling approach is, given a single ground-level RGB image, to obtain a global geolocation probability map that incorporates information from base geolocation models (in this work, GeoEstimation and GeoCLIP) and ground-level maps obtained from satellite data products. The image is passed to both the base geolocation model and the individual ground-level attribute predictors. Example output $S^2$ cell and ground level attribute probabilities from Algo. \ref{['algo:geoest']} and \ref{['algo:model_isec']}, and final ensembled probabilities are shown. The high probability area is significantly reduced in ensembling. Probability maps thresholded for visualization.
• Figure 2: Batches of images from (left) MP-16 and (right) Street View, from medium urban density regions per LandScan labeling (see Sec. \ref{['sec:datasets']}, \ref{['sec:predictors']}). In general, the MP-16 dataset consists of more typical urban scenes, whereas Street View contains more scenery from near roads. Further MP-16 dataset is drawn from areas much more close to urban environments.
• Figure 3: Example overlay centered on Europe, showing $S^2$ cells (red boxes, fine level from GeoEstimation muller2018geolocation) with LandScansims2022landscan medium population density mask (white). The mask is much more restrictive than the $S^2$ cells alone, yielding improvements in RvA from model ensembling.
• Figure 4: (a) Recall vs. Area ( RvA) obtained for rasterized GeoEst (Alg.\ref{['algo:geoest']}), alone and ensembled with LandScan (+LS) attribute prediction. As an additional baseline, we compare always applying the LS "urban" mask instead of predicting the bucket (+urban, dotted curve). Spherical cap areas shown in blacked dashed lines, see Sec. \ref{['sec:results']} for details. (b) RvA, measured on data rebalanced over urban and non-urban areas by randomly sampling equal number of images from each LS mask. Our method improves performance for less-populated areas while maintaining urban area results. (c) Relative improvement measured on the balanced datasets.
• Figure 5: Rasterized GeoCLIP \ref{['algo:geoest']} and ensembling: same analysis as Fig. \ref{['fig:geoest_recall_vs_area']}(a--c).