Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning

Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Yong Li, Pan Hui

TL;DR

This work tackles the challenge of predicting urban socio-economic status from visual data with strong generalization to unseen cities and indicators. It introduces CityRiSE, a reinforcement-learning framework built on large vision-language models, guided by verifiable rewards and enriched with perceptual and general visual reasoning data to evoke emergent, interpretable reasoning. A Transferable Dataset and a GRPO-based Training Pipeline underpin the approach, with Keyword and Regression rewards aligning perceptual grounding and numeric accuracy. Experiments across 11 SES tasks show CityRiSE outperforms baselines and generalizes across cities and indicators, delivering interpretable reasoning traces that aid trust and applicability in urban planning contexts.

Abstract

Harnessing publicly available, large-scale web data, such as street view and satellite imagery, urban socio-economic sensing is of paramount importance for achieving global sustainable development goals. With the emergence of Large Vision-Language Models (LVLMs), new opportunities have arisen to solve this task by treating it as a multi-modal perception and understanding problem. However, recent studies reveal that LVLMs still struggle with accurate and interpretable socio-economic predictions from visual data. To address these limitations and maximize the potential of LVLMs, we introduce \textbf{CityRiSE}, a novel framework for \textbf{R}eason\textbf{i}ng urban \textbf{S}ocio-\textbf{E}conomic status in LVLMs through pure reinforcement learning (RL). With carefully curated multi-modal data and verifiable reward design, our approach guides the LVLM to focus on semantically meaningful visual cues, enabling structured and goal-oriented reasoning for generalist socio-economic status prediction. Experiments demonstrate that CityRiSE with emergent reasoning process significantly outperforms existing baselines, improving both prediction accuracy and generalization across diverse urban contexts, particularly for prediction on unseen cities and unseen indicators. This work highlights the promise of combining RL and LVLMs for interpretable and generalist urban socio-economic sensing.

CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning

TL;DR

This work tackles the challenge of predicting urban socio-economic status from visual data with strong generalization to unseen cities and indicators. It introduces CityRiSE, a reinforcement-learning framework built on large vision-language models, guided by verifiable rewards and enriched with perceptual and general visual reasoning data to evoke emergent, interpretable reasoning. A Transferable Dataset and a GRPO-based Training Pipeline underpin the approach, with Keyword and Regression rewards aligning perceptual grounding and numeric accuracy. Experiments across 11 SES tasks show CityRiSE outperforms baselines and generalizes across cities and indicators, delivering interpretable reasoning traces that aid trust and applicability in urban planning contexts.

Abstract

Harnessing publicly available, large-scale web data, such as street view and satellite imagery, urban socio-economic sensing is of paramount importance for achieving global sustainable development goals. With the emergence of Large Vision-Language Models (LVLMs), new opportunities have arisen to solve this task by treating it as a multi-modal perception and understanding problem. However, recent studies reveal that LVLMs still struggle with accurate and interpretable socio-economic predictions from visual data. To address these limitations and maximize the potential of LVLMs, we introduce \textbf{CityRiSE}, a novel framework for \textbf{R}eason\textbf{i}ng urban \textbf{S}ocio-\textbf{E}conomic status in LVLMs through pure reinforcement learning (RL). With carefully curated multi-modal data and verifiable reward design, our approach guides the LVLM to focus on semantically meaningful visual cues, enabling structured and goal-oriented reasoning for generalist socio-economic status prediction. Experiments demonstrate that CityRiSE with emergent reasoning process significantly outperforms existing baselines, improving both prediction accuracy and generalization across diverse urban contexts, particularly for prediction on unseen cities and unseen indicators. This work highlights the promise of combining RL and LVLMs for interpretable and generalist urban socio-economic sensing.
Paper Structure (35 sections, 4 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 35 sections, 4 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Transferable Datasets
  4. Socio-Economic Indicator Data
  5. Perceptual Urban Reasoning Data
  6. Spatial Reasoning
  7. Geolocation Task
  8. Socio-Economic Ranking
  9. General Visual Reasoning Data
  10. Verifiable Reward
  11. Keyword Reward
  12. Regression Reward
  13. Training Pipeline
  14. Experiments
  15. Settings
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Framework of CityRiSE, which integrates three types of training data with a GRPO training pipeline. The GRPO algorithm incorporates both Keyword Reward and Regression Reward to guide learning.
  • Figure 2: Results of Reward Ablation, where 'Reg.' refers to the Regression Reward, 'Key.' refers to the Keyword Reward, and 'w/o Both' indicates the setting where both rewards are removed.
  • Figure 3: Results of Data Ablation, where 'Per.' refers to the Perceptual Urban Reasoning Data, 'Gen.' refers to the General Visual Reasoning Data.
  • Figure 4: Qualitative comparison of reasoning patterns produced by different models. CityRiSE exhibits more interpretable and goal-directed reasoning behavior, while baseline models often generate less informative responses.