GPS as a Control Signal for Image Generation

TL;DR

This work demonstrates that GPS metadata can serve as a powerful conditioning signal for image generation, enabling location-aware synthesis that captures fine-grained urban variations. By training GPS-conditioned diffusion models on geotagged photos and combining GPS with text prompts, the approach achieves compositional generation and can produce location-specific imagery across cities. The authors extend this framework to 3D by lifting NeRFs from 2D GPS-conditioned models through score distillation sampling, incorporating view-dependent cues via angle conditioning to improve geometric fidelity. Experiments across New York and Paris show superior GPS- and text-conditioned generation over baselines and reveal robust 3D reconstructions where traditional SfM pipelines struggle. The work suggests GPS conditioning as a complementary tool for geospatial analysis of imagery and for extracting structured 3D information from large, unconstrained photo collections.

Abstract

We show that the GPS tags contained in photo metadata provide a useful control signal for image generation. We train GPS-to-image models and use them for tasks that require a fine-grained understanding of how images vary within a city. In particular, we train a diffusion model to generate images conditioned on both GPS and text. The learned model generates images that capture the distinctive appearance of different neighborhoods, parks, and landmarks. We also extract 3D models from 2D GPS-to-image models through score distillation sampling, using GPS conditioning to constrain the appearance of the reconstruction from each viewpoint. Our evaluations suggest that our GPS-conditioned models successfully learn to generate images that vary based on location, and that GPS conditioning improves estimated 3D structure.

Figures (15)

• Figure 1: What can we do with a GPS-conditioned image generation model? We train GPS-to-image models and use them for tasks that require a fine-grained understanding of how images vary within a city. For example, a model trained on densely sampled geotagged photos from Manhattan can generate images that match a neighborhood's general appearance and capture key landmarks like museums and parks. We show images sampled from a variety of GPS locations and text prompts. For example, an image with the text prompt "bagel" results in a modern-style sculpture when conditioned on the Museum of Modern Art and an impressionist-style painting when conditioned on the Metropolitan Museum of Art. We also "lift" a 3D NeRF of the Statue of Liberty from a landmark-specific 2D GPS-to-image model using score distillation sampling. Please see the https://cfeng16.github.io/gps-gen/ and Sec. \ref{['supp:more_qua_results']} for more examples.
• Figure 2: Method. (a) After downloading geotagged photos, we train a GPS-to-image generation model conditioned on GPS tags and text prompts. The trained generative model can produce images using both conditioning signals in a compositional manner. (b) We can also extract 3D models from a landmark-specific GPS-to-image model using score distillation sampling. This diffusion model parameterizes the GPS location by the azimuth with respect to a given landmark's center. + means we concatenate GPS embeddings and text embeddings.
• Figure 3: 3D Setup Comparison. We extract 3D models from 2D GPS-to-image models. (a) Traditional approaches require running SfM to estimate camera pose, followed by dense geometry estimation. Since they are based on triangulation, they are susceptible to catastrophic errors due to incorrect pose; (b) DreamFusion poole2023dreamfusion samples images from different poses within a scene using view-dependent prompting. However, text has a limited ability to precisely control the position of the camera. (c) Our method extends DreamFusion with GPS conditioning, reducing pose uncertainty.
• Figure 4: Qualitative results for Paris. We show images that have been sampled from our GPS-to-image diffusion model for various locations and prompts within Paris.
• Figure 5: Qualitative comparison for GPS-to-image diffusion. We compare the qualitative results of our method against baselines using specific pairs of text prompts and GPS tags. Each column shows a text prompt and a GPS tag at the top. Text-address-to-image diffusion model is conditioned on a combination of the text prompt and the address name derived from the GPS tag. We also perform nearest neighbor in the training set based on GPS tags. Our GPS-to-image diffusion model uses a text prompt and raw GPS tag as conditioning. Google Street View images are sampled for reference of geolocation. Our method achieves better compositionality and visual quality.