Into the Unknown: Generating Geospatial Descriptions for New Environments

TL;DR

This work tackles the data-scarce problem of geospatial instruction following in new environments by building a grounded OpenStreetMap-based knowledge graph and generating synthetic, spatially grounded instructions through two routes: a rule-based CFG framework and prompting large language models. The CFG-based augmentation yields superior performance over LLM-based methods on the Rendezvous (RVS) task, particularly in unseen environments, achieving substantial gains in 100 m accuracy and reductions in mean/median distance errors, and narrowing the human-AI gap. The study also analyzes data quantity vs. quality trade-offs, showing that large synthetic datasets can outperform smaller, human-annotated sets and that combining synthetic with real data provides further improvements, while highlighting artifacts and hallucinations in LLM-generated instructions. Overall, explicitly structuring spatial relations via CFG templates offers interpretable, scalable gains for text-based geospatial reasoning in data-scarce scenarios, with future work aiming to integrate visual cues and further mitigate artifacts from generation models.

Abstract

Similar to vision-and-language navigation (VLN) tasks that focus on bridging the gap between vision and language for embodied navigation, the new Rendezvous (RVS) task requires reasoning over allocentric spatial relationships (independent of the observer's viewpoint) using non-sequential navigation instructions and maps. However, performance substantially drops in new environments with no training data. Using opensource descriptions paired with coordinates (e.g., Wikipedia) provides training data but suffers from limited spatially-oriented text resulting in low geolocation resolution. We propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. Our method constructs a grounded knowledge-graph, capturing entity relationships. Sampled entities and relations (`shop north of school') generate navigation instructions via (i) generating numerous templates using context-free grammar (CFG) to embed specific entities and relations; (ii) feeding the entities and relation into a large language model (LLM) for instruction generation. A comprehensive evaluation on RVS, showed that our approach improves the 100-meter accuracy by 45.83% on unseen environments. Furthermore, we demonstrate that models trained with CFG-based augmentation achieve superior performance compared with those trained with LLM-based augmentation, both in unseen and seen environments. These findings suggest that the potential advantages of explicitly structuring spatial information for text-based geospatial reasoning in previously unknown, can unlock data-scarce scenarios.

Figures (5)

• Figure 1: Our method for generating spatial descriptions samples from the graph-map (top) a path (middle image, red line), a starting point (green marker), a goal point (red marker), and prominent landmarks (black markers). It then generates an instruction (bottom) from the spatial relations between these entities.
• Figure 2: Instruction generation steps (for example presented in Figure \ref{['fig:main_example']}): (i) Template creation via CFG; (ii) replacing generic elements (in capital letters) with specific landmarks and spatial relations (above the lines).
• Figure 3: T5 performance (Y-axis) with varying AUG-CFG training samples (X-axis). (a,b) RVS seen-city (Manhattan), (c,d) RVS unseen-city (Pittsburgh).
• Figure 4: Cumulative distribution function (CDF) error. Augmentation impact on distance error (meters).
• Figure 5: The RVS model based on a T5 transformer and a graph representation of the environment RVS.