Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, Yuxuan Liang

TL;DR

This work introduces the first-ever LLM-enhanced framework that integrates the knowledge of text modality into urban imagery, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP), and generates a detailed textual description for each satellite image by Image-to-Text LLMs.

Abstract

Urban region profiling from web-sourced data is of utmost importance for urban planning and sustainable development. We are witnessing a rising trend of LLMs for various fields, especially dealing with multi-modal data research such as vision-language learning, where the text modality serves as a supplement information for the image. Since textual modality has never been introduced into modality combinations in urban region profiling, we aim to answer two fundamental questions in this paper: i) Can textual modality enhance urban region profiling? ii) and if so, in what ways and with regard to which aspects? To answer the questions, we leverage the power of Large Language Models (LLMs) and introduce the first-ever LLM-enhanced framework that integrates the knowledge of textual modality into urban imagery profiling, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP). Specifically, it first generates a detailed textual description for each satellite image by an open-source Image-to-Text LLM. Then, the model is trained on the image-text pairs, seamlessly unifying natural language supervision for urban visual representation learning, jointly with contrastive loss and language modeling loss. Results on predicting three urban indicators in four major Chinese metropolises demonstrate its superior performance, with an average improvement of 6.1% on R^2 compared to the state-of-the-art methods. Our code and the image-language dataset will be released upon paper notification.

UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

TL;DR

This work introduces the first-ever LLM-enhanced framework that integrates the knowledge of text modality into urban imagery, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP), and generates a detailed textual description for each satellite image by Image-to-Text LLMs.

Abstract

Urban region profiling from web-sourced data is of utmost importance for urban planning and sustainable development. We are witnessing a rising trend of LLMs for various fields, especially dealing with multi-modal data research such as vision-language learning, where the text modality serves as a supplement information for the image. Since textual modality has never been introduced into modality combinations in urban region profiling, we aim to answer two fundamental questions in this paper: i) Can textual modality enhance urban region profiling? ii) and if so, in what ways and with regard to which aspects? To answer the questions, we leverage the power of Large Language Models (LLMs) and introduce the first-ever LLM-enhanced framework that integrates the knowledge of textual modality into urban imagery profiling, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP). Specifically, it first generates a detailed textual description for each satellite image by an open-source Image-to-Text LLM. Then, the model is trained on the image-text pairs, seamlessly unifying natural language supervision for urban visual representation learning, jointly with contrastive loss and language modeling loss. Results on predicting three urban indicators in four major Chinese metropolises demonstrate its superior performance, with an average improvement of 6.1% on R^2 compared to the state-of-the-art methods. Our code and the image-language dataset will be released upon paper notification.
Paper Structure (37 sections, 6 equations, 8 figures, 2 tables)

This paper contains 37 sections, 6 equations, 8 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. Formulation
  4. Related Work
  5. Urban Region Profiling
  6. Large Language Model
  7. METHODOLOGY
  8. Text Generation and Refinement
  9. Single-modality Representation Learning
  10. Cross-modality Representation Learning
  11. Urban Indicator Prediction
  12. Discussion
  13. Additional Data Alignment and Integration
  14. Prompt-guided Downstream Tasks
  15. EXPERIMENTS
  16. ...and 22 more sections

Figures (8)

  • Figure 1: Frameworks for urban region profiling. We present the first attempt to leverage the power of LLMs for this task.
  • Figure 2: Overall framework of the proposed UrbanCLIP.
  • Figure 3: Text generation and refinement.
  • Figure 4: Results of Ablation Study on $R^2$ Metric.
  • Figure 5: $R^2$ heatmap for the transferability test between UrbanCLIP and PG-SimCLR, on 3 urban indicators across 4 datasets.
  • ...and 3 more figures