GaGA: Towards Interactive Global Geolocation Assistant

TL;DR

This work advances geographic localization by introducing GaGA, an interactive global geolocation assistant built on multimodal vision-language models. It defines an interactive paradigm that refines location predictions through user input and explanations, supported by the MG-Geo dataset with 5 million image-text pairs and a three-part design for perception, clues, and dialogue. GaGA achieves state-of-the-art performance on benchmark data while delivering interpretable clues and justifications, enabling real-time, user-guided corrections. The contributions include the MG-Geo dataset, the three-mode GaGA system, and extensive experiments showing improved country- and city-level accuracy, enhanced interactive reasoning, and robust dialog quality, with implications for practical, privacy-aware geolocation applications.

Abstract

Global geolocation, which seeks to predict the geographical location of images captured anywhere in the world, is one of the most challenging tasks in the field of computer vision. In this paper, we introduce an innovative interactive global geolocation assistant named GaGA, built upon the flourishing large vision-language models (LVLMs). GaGA uncovers geographical clues within images and combines them with the extensive world knowledge embedded in LVLMs to determine the geolocations while also providing justifications and explanations for the prediction results. We further designed a novel interactive geolocation method that surpasses traditional static inference approaches. It allows users to intervene, correct, or provide clues for the predictions, making the model more flexible and practical. The development of GaGA relies on the newly proposed Multi-modal Global Geolocation (MG-Geo) dataset, a comprehensive collection of 5 million high-quality image-text pairs. GaGA achieves state-of-the-art performance on the GWS15k dataset, improving accuracy by 4.57% at the country level and 2.92% at the city level, setting a new benchmark. These advancements represent a significant leap forward in developing highly accurate, interactive geolocation systems with global applicability.

Figures (5)

• Figure 1: The mainstream Global Geolocation models. GaGA, trained with the MG-Geo dataset, advances traditional black-box localization models to a new paradigm of interactive geolocation.
• Figure 2: The architecture of GaGA, comprising a vision encoder $f_{VM}$, a projector layer $f_P$, a text tokenizer $f_T$, and a large language model $f_L$.
• Figure 3: Illustrations of GaGA's dialogues in various scenarios. On the left, we demonstrate how GaGA successfully incorporates external knowledge with human guidance; on the right, we showcase the model's predictive outcomes when given relevant prior information.
• Figure 4: Data Statistics of Geo-Localized Question-Answer Clue Pairs and Clue Generation Pipeline. (a) The pipeline for generating image-clue pairs. (b) Statistical data on clue types, categorized into eight major categories covering a wide range of topics. (c) Country cue pairs for the Dialog Part, where the inner circle represents the countries in the output responses, and the outer circle represents the direct clues.
• Figure 5: Distribution of GWS15k. For detailed information on the reproduction algorithm, please refer to Section\ref{['gws15k_reproduction']}.