Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geography According to ChatGPT -- How Generative AI Represents and Reasons about Geography

Krzysztof Janowicz, Gengchen Mai, Rui Zhu, Song Gao, Zhangyu Wang, Yingjie Hu, Lauren Bennett

Abstract

Understanding how AI will represent and reason about geography should be a key concern for all of us, as the broader public increasingly interacts with spaces and places through these systems. Similarly, in line with the nature of foundation models, our own research often relies on pre-trained models. Hence, understanding what world AI systems construct is as important as evaluating their accuracy, including factual recall. To motivate the need for such studies, we provide three illustrative vignettes, i.e., exploratory probes, in the hope that they will spark lively discussions and follow-up work: (1) Do models form strong defaults, and how brittle are model outputs to minute syntactic variations? (2) Can distributional shifts resurface from the composition of individually benign tasks, e.g., when using AI systems to create personas? (3) Do we overlook deeper questions of understanding when solely focusing on the ability of systems to recall facts such as geographic principles?

Geography According to ChatGPT -- How Generative AI Represents and Reasons about Geography

Abstract

Understanding how AI will represent and reason about geography should be a key concern for all of us, as the broader public increasingly interacts with spaces and places through these systems. Similarly, in line with the nature of foundation models, our own research often relies on pre-trained models. Hence, understanding what world AI systems construct is as important as evaluating their accuracy, including factual recall. To motivate the need for such studies, we provide three illustrative vignettes, i.e., exploratory probes, in the hope that they will spark lively discussions and follow-up work: (1) Do models form strong defaults, and how brittle are model outputs to minute syntactic variations? (2) Can distributional shifts resurface from the composition of individually benign tasks, e.g., when using AI systems to create personas? (3) Do we overlook deeper questions of understanding when solely focusing on the ability of systems to recall facts such as geographic principles?
Paper Structure (5 sections, 1 equation, 3 figures)

This paper contains 5 sections, 1 equation, 3 figures.

Table of Contents

  1. Introduction and Motivation
  2. Vignette I: Default Strength and Brittleness
  3. Vignette II: Beyond Surface Bias
  4. Vignette III: Model Knows, Model Shows
  5. Conclusions and Outlook

Figures (3)

  • Figure 1: Results for the GPT family for the 'Name a country, please.' prompt at a higher temperature of 0.7 (without reasoning for 5.1) to force more varied outputs.
  • Figure 2: Criminal record labels in the fictitious book population compared to pre-COVID arrest data from the LAPD. Note, the difference between those distributions as such does not imply any racial bias.
  • Figure 3: Cities ranked by population: US versus the fictitious island nation of Novaterra.