Urban Socio-Semantic Segmentation with Vision-Language Reasoning

TL;DR

This work tackles urban socio-semantic segmentation by introducing SocioSeg, a hierarchical, map-aligned satellite dataset, and SocioReasoner, a two-stage vision-language reasoning framework that renders and refines prompts through a render-and-refine loop. It leverages reinforcement learning with GRPO to optimize a non-differentiable reasoning process, enabling open-vocabulary social-category segmentation that surpasses baselines and shows strong zero-shot generalization to unseen map styles and regions. The combination of digital map context with satellite imagery and iterative reasoning significantly improves segmentation of socially defined entities like schools and parks, with broad implications for urban planning and governance. Overall, the approach demonstrates the value of cross-modal reasoning and interactive prompting in complex geospatial analysis tasks.

Abstract

As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in https://github.com/AMAP-ML/SocioReasoner.

Figures (14)

• Figure 1: (a) Current works segregate physical entities. (b) Our SocioSeg identifies social entities (names, functions) via multi-modal data. (c) Existing reasoning methods employ a single-stage reasoning approach. (d) Our SocioReasoner employs a two-stage reasoning strategy with render-and-refine mechanism.
• Figure 2: SocioReasoner Framework. Given a satellite image, a digital map, and a textual instruction, the VLM first generates bounding boxes to localize candidate regions. These boxes are fed into SAM to produce a coarse mask. The boxes and mask are then rendered onto the inputs for re-evaluation. The VLM emits boxes and points, which are again fed into SAM to yield the final mask.
• Figure 3: Visualization of the SocioReasoner results. The top panel shows a comparison between the results of SocioReasoner (with both stages visualized) and competitive baselines. The bottom-left panel illustrates the reasoning process of SocioReasoner. The bottom-right panel displays the visualization results of SocioReasoner on the out-of-domain dataset.
• Figure 4: Per-class accuracy comparison across Socio-classes. We select the top-20 most frequent classes in the test set for visualization. The full results are available in Appendix \ref{['appendix:per_categories_results']}.
• Figure 5: (a) Sum reward during training. It shows the sum of rewards across training steps in the two-stage workflow. (b) Multi-stage gIoU during training. It shows the gIoU improvement across training steps in the two-stage workflow. (c) Different number of points. It visualizes the result of SocioReasoner in the refinement stage with different numbers of points.