Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Research

Hamid Sarmadi, Ola Hall, Thorsteinn Rögnvaldsson, Mattias Ohlsson

TL;DR

The paper addresses village-level poverty estimation from satellite imagery using vision-enabled LLMs. It uses a pairwise comparison framework with GPT-4o across 608 DHS sites, generating 184,528 image-pair judgments, and aggregates results with an Iterative Luce Spectral Ranking Bradley-Terry approach to produce a global wealth ranking. CNN-based wealth estimation from Sentinel-2 imagery achieves the highest rank correlation, ρ ≈ 0.78, while ChatGPT achieves ρ ≈ 0.56, showing scalability and competitive performance without task-specific fine-tuning. The study highlights both promise and limitations of multimodal LLMs for socioeconomic monitoring and suggests workflow improvements via prompt engineering and richer multimodal context. It contributes to the broader exploration of unconventional data sources for welfare analysis and demonstrates potential for cost-effective, large-scale poverty monitoring.

Abstract

This paper investigates the novel application of Large Language Models (LLMs) with vision capabilities to analyze satellite imagery for village-level poverty prediction. Although LLMs were originally designed for natural language understanding, their adaptability to multimodal tasks, including geospatial analysis, has opened new frontiers in data-driven research. By leveraging advancements in vision-enabled LLMs, we assess their ability to provide interpretable, scalable, and reliable insights into human poverty from satellite images. Using a pairwise comparison approach, we demonstrate that ChatGPT can rank satellite images based on poverty levels with accuracy comparable to domain experts. These findings highlight both the promise and the limitations of LLMs in socioeconomic research, providing a foundation for their integration into poverty assessment workflows. This study contributes to the ongoing exploration of unconventional data sources for welfare analysis and opens pathways for cost-effective, large-scale poverty monitoring.

Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Research

TL;DR

The paper addresses village-level poverty estimation from satellite imagery using vision-enabled LLMs. It uses a pairwise comparison framework with GPT-4o across 608 DHS sites, generating 184,528 image-pair judgments, and aggregates results with an Iterative Luce Spectral Ranking Bradley-Terry approach to produce a global wealth ranking. CNN-based wealth estimation from Sentinel-2 imagery achieves the highest rank correlation, ρ ≈ 0.78, while ChatGPT achieves ρ ≈ 0.56, showing scalability and competitive performance without task-specific fine-tuning. The study highlights both promise and limitations of multimodal LLMs for socioeconomic monitoring and suggests workflow improvements via prompt engineering and richer multimodal context. It contributes to the broader exploration of unconventional data sources for welfare analysis and demonstrates potential for cost-effective, large-scale poverty monitoring.

Abstract

This paper investigates the novel application of Large Language Models (LLMs) with vision capabilities to analyze satellite imagery for village-level poverty prediction. Although LLMs were originally designed for natural language understanding, their adaptability to multimodal tasks, including geospatial analysis, has opened new frontiers in data-driven research. By leveraging advancements in vision-enabled LLMs, we assess their ability to provide interpretable, scalable, and reliable insights into human poverty from satellite images. Using a pairwise comparison approach, we demonstrate that ChatGPT can rank satellite images based on poverty levels with accuracy comparable to domain experts. These findings highlight both the promise and the limitations of LLMs in socioeconomic research, providing a foundation for their integration into poverty assessment workflows. This study contributes to the ongoing exploration of unconventional data sources for welfare analysis and opens pathways for cost-effective, large-scale poverty monitoring.
Paper Structure (18 sections, 4 figures, 3 tables)

This paper contains 18 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Survey Data (DHS Wealth Index)
  4. Satellite Imagery
  5. Methods
  6. Pairwise comparison of images
  7. Producing the final ranking
  8. Convolutional Neural Network (CNN)
  9. Random Forest regressor
  10. Evaluation measures
  11. Results
  12. Developing the prompt
  13. From pair-wise comparisons to the overall ranking
  14. The quality of the rankings
  15. Discussion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Examples of medium resolution images ($2 \times 2$ km) used in training the CNN (left) compared to their corresponding high resolution images shown to ChatGPT (right). The area for the high resolution image is marked with a red square in the corresponding medium resolution image.
  • Figure 2: Upper left panel: Histogram of the differences in HV271 rankings for the pairs where the ChatGPT pairwise comparison disagrees with what would be expected from HV271 (i.e. when the wealthier/poorer comparison has opposite signs in the HV271 and the ChatGPT comparison). Upper right and lower panels: Histograms of the differences in rankings between pairs in the final ChatGPT ranking and the pairwise comparison matrix; the panels show the results for the three algorithms tested to infer the overall ranking from the pairwise comparison matrix. Compare with numbers in Table \ref{['tab:ranking_inference_results']}.
  • Figure 3: Scatter plots showing the ranked wealth predictions versus the ranked true wealth (averages of HV271) for the 608 DHS clusters. The $\rho$ value is the Spearman's rank correlation.
  • Figure 4: Examples of images related to different points where the ranking obtained from ChatGPT disagrees from the one from the HV271 wealthindex.