Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing

Tianhui Liu, Hetian Pang, Xin Zhang, Tianjian Ouyang, Zhiyuan Zhang, Jie Feng, Yong Li, Pan Hui

TL;DR

CityLens is introduced, a comprehensive benchmark designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery, and reveals that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators.

Abstract

Understanding urban socioeconomic conditions through visual data is a challenging yet essential task for sustainable urban development and policy planning. In this work, we introduce \textit{CityLens}, a comprehensive benchmark designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery. We construct a multi-modal dataset covering a total of 17 globally distributed cities, spanning 6 key domains: economy, education, crime, transport, health, and environment, reflecting the multifaceted nature of urban life. Based on this dataset, we define 11 prediction tasks and utilize 3 evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. We benchmark 17 state-of-the-art LVLMs across these tasks. These make CityLens the most extensive socioeconomic benchmark to date in terms of geographic coverage, indicator diversity, and model scale. Our results reveal that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators. CityLens provides a unified framework for diagnosing these limitations and guiding future efforts in using LVLMs to understand and predict urban socioeconomic patterns. The code and data are available at https://github.com/tsinghua-fib-lab/CityLens.

CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing

TL;DR

CityLens is introduced, a comprehensive benchmark designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery, and reveals that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators.

Abstract

Understanding urban socioeconomic conditions through visual data is a challenging yet essential task for sustainable urban development and policy planning. In this work, we introduce \textit{CityLens}, a comprehensive benchmark designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery. We construct a multi-modal dataset covering a total of 17 globally distributed cities, spanning 6 key domains: economy, education, crime, transport, health, and environment, reflecting the multifaceted nature of urban life. Based on this dataset, we define 11 prediction tasks and utilize 3 evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. We benchmark 17 state-of-the-art LVLMs across these tasks. These make CityLens the most extensive socioeconomic benchmark to date in terms of geographic coverage, indicator diversity, and model scale. Our results reveal that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators. CityLens provides a unified framework for diagnosing these limitations and guiding future efforts in using LVLMs to understand and predict urban socioeconomic patterns. The code and data are available at https://github.com/tsinghua-fib-lab/CityLens.

Paper Structure

This paper contains 65 sections, 14 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Dataset Construction
  4. Data Collection
  5. Indicator Selection
  6. Data Mapping
  7. Evaluation Methodologies
  8. Direct Metric Prediction
  9. Normalized Metric Estimation
  10. Feature-Based Regression
  11. Experiments
  12. Overall Performance on Feature-Based Regression
  13. Challenge of the Benchmark for LVLMs
  14. Performance Differences Across Models
  15. Variations Across Different Task Types
  16. ...and 50 more sections

Figures (14)

  • Figure 1: CityLens comprises 11 socioeconomic indicators spanning six key urban domains.
  • Figure 2: Benchmark Construction Pipeline, including data collection, indicator selection, data mapping and evaluation methods.
  • Figure 3: (a) 11 indicators in benchmark and their counts. (b) Statistics of dataset.
  • Figure 4: Prompt examples for three evaluation methodologies.
  • Figure 5: Comparison of task-wise $R^2$ performance between Direct Metric Prediction and Normalized Estimation across 11 socioeconomic indicators in CityLens.
  • ...and 9 more figures