Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What can LLM tell us about cities?

Zhuoheng Li, Yaochen Wang, Zhixue Song, Yuqi Huang, Rui Bao, Guanjie Zheng, Zhenhui Jessie Li

TL;DR

It is observed that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks.

Abstract

This study explores the capabilities of large language models (LLMs) in providing knowledge about cities and regions on a global scale. We employ two methods: directly querying the LLM for target variable values and extracting explicit and implicit features from the LLM correlated with the target variable. Our experiments reveal that LLMs embed a broad but varying degree of knowledge across global cities, with ML models trained on LLM-derived features consistently leading to improved predictive accuracy. Additionally, we observe that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks. These findings suggest that LLMs can offer new opportunities for data-driven decision-making in the study of cities.

What can LLM tell us about cities?

TL;DR

It is observed that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks.

Abstract

This study explores the capabilities of large language models (LLMs) in providing knowledge about cities and regions on a global scale. We employ two methods: directly querying the LLM for target variable values and extracting explicit and implicit features from the LLM correlated with the target variable. Our experiments reveal that LLMs embed a broad but varying degree of knowledge across global cities, with ML models trained on LLM-derived features consistently leading to improved predictive accuracy. Additionally, we observe that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks. These findings suggest that LLMs can offer new opportunities for data-driven decision-making in the study of cities.

Paper Structure

This paper contains 99 sections, 9 figures, 45 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM as a Knowledge Base
  4. Extracting Knowledge from LLM
  5. LLM for City Tasks
  6. Problem and Method
  7. Directly Asking LLM
  8. Implicit Feature Extraction
  9. Explicit Feature Extraction
  10. Experiment
  11. Experiment Settings
  12. Overall Results
  13. Q1: Can we determine when LLM knows or does not know the answer?
  14. Detecting consistent generic values.
  15. Detecting inconsistent values among different rounds of generations.
  16. ...and 84 more sections

Figures (9)

  • Figure 1: Taxi pick-up and drop-off patterns in NYC: the commercial Garment District (left) shows morning drop-off and evening pick-up peaks, while the residential Arden Heights (right) shows opposite trends.
  • Figure 2: Public transportation data are collected from various agencies. Source: verbavatz2020access.
  • Figure 3: A strong correlation between public transportation scores retrieved from LLM and the public transportation metric computed from raw transportation data verbavatz2020access.
  • Figure 4: Features of a specific task are extracted from the last hidden state using Llama3.1-8B.
  • Figure 5: Extract relevant features impacting city-level energy consumption and provide their scaled values for a specific location (e.g. New York City in 2024) to better understand the factors influencing energy usage.
  • ...and 4 more figures