EarthLoc: Astronaut Photography Localization by Indexing Earth from Space

TL;DR

This work introduces innovative training techniques which contribute to the development of a high-performance model, EarthLoc, and performs a comprehensive benchmark comparing EarthLoc to existing methods, showcasing its superior efficiency and accuracy.

Abstract

Astronaut photography, spanning six decades of human spaceflight, presents a unique Earth observations dataset with immense value for both scientific research and disaster response. Despite its significance, accurately localizing the geographical extent of these images, crucial for effective utilization, poses substantial challenges. Current manual localization efforts are time-consuming, motivating the need for automated solutions. We propose a novel approach - leveraging image retrieval - to address this challenge efficiently. We introduce innovative training techniques, including Year-Wise Data Augmentation and a Neutral-Aware Multi-Similarity Loss, which contribute to the development of a high-performance model, EarthLoc. We develop six evaluation datasets and perform a comprehensive benchmark comparing EarthLoc to existing methods, showcasing its superior efficiency and accuracy. Our approach marks a significant advancement in automating the localization of astronaut photography, which will help bridge a critical gap in Earth observations data. Code and datasets are available at https://github.com/gmberton/EarthLoc

Figures (15)

• Figure 1: Overview of the astronaut photography localization task. Astronauts take hundreds of photos a day from the International Space Station (ISS) cupola (top-left) with hand-held cameras. For each image (example bottom right), the geographic location depicted is not known, and needs to be searched for across a huge area centered at the ISS's (known) nadir point at the moment that the photo is taken. A simulated view of the astronaut's perspective when the ISS is above Europe is shown. The goal of our paper is to automate the task of localizing these images, which could be anywhere within the view. In the figure's example, the photo the astronaut took is indicated by the green line and shown in inset -- other possible photo extents are in red, illustrating the wide array of potential locations to search.
• Figure 2: Astronaut photo query examples, showcasing the large variability in covered area and appearance.
• Figure 3: Examples of database images.Top: four images from different regions, with 25% or 50% overlap between any pair. The red dot in each image represents the same geographic point. Bottom: 4 images from the same region across different years.
• Figure 4: To create evaluation sets we choose all images that could contain a Point of Interest (POI) - all photos with nadir point within $d_{visible}$. To localize all photos within this range, even if they do not contain the POI, we create a database that contains all areas visible from the nadir points of all selected photos, yielding a database area of about 5000km$^2$ per POI.
• Figure 5: Training strategy. Naive batching produces too easy a task for the model to learn robust representations from. We increase difficulty by clustering batches by similarity and adding year-wise augmentations, which move images of different regions, but from the same year, closer together in feature space. The model must then learn representations that discern similarity in this more difficult context.