WalkCLIP: Multimodal Learning for Urban Walkability Prediction

TL;DR

WalkCLIP presents a three-stage multimodal framework that combines street-view and satellite imagery with population dynamics to predict urban walkability. By finetuning CLIP models with GPT-4o-generated captions, applying a Spatially-Aware Feature Enhancement module, and fusing visual features with PDFM context, the approach achieves superior predictive accuracy and spatial coherence (R^2 = 0.887) over MSP locations compared with unimodal and other multimodal baselines. The work demonstrates the value of integrating visual and behavioral signals for scalable, neighborhood-level walkability assessment and highlights opportunities to extend temporal analysis and generalize to more cities. Practically, WalkCLIP offers a scalable tool for planners and researchers to monitor and compare walking environments across urban areas.

Abstract

Urban walkability is a cornerstone of public health, sustainability, and quality of life. Traditional walkability assessments rely on surveys and field audits, which are costly and difficult to scale. Recent studies have used satellite imagery, street view imagery, or population indicators to estimate walkability, but these single-source approaches capture only one dimension of the walking environment. Satellite data describe the built environment from above, but overlook the pedestrian perspective. Street view imagery captures conditions at the ground level, but lacks broader spatial context. Population dynamics reveal patterns of human activity but not the visual form of the environment. We introduce WalkCLIP, a multimodal framework that integrates these complementary viewpoints to predict urban walkability. WalkCLIP learns walkability-aware vision-language representations from GPT-4o generated image captions, refines these representations with a spatial aggregation module that incorporates neighborhood context, and fuses the resulting features with representations from a population dynamics foundation model. Evaluated at 4,660 locations throughout Minneapolis-Saint Paul, WalkCLIP outperforms unimodal and multimodal baselines in both predictive accuracy and spatial alignment. These results show that the integration of visual and behavioral signals yields reliable predictions of the walking environment.

Figures (6)

• Figure 1: The WalkCLIP framework with three stages. (1) Visual representations are learned by finetuning a CLIP transformer on satellite and street view images paired with GPT-4o-generated captions. (2) The Spatially-Aware Feature Enhancement (SAFE) module builds a proximity graph and aggregates features from nearby locations to produce spatially-aware representations. (3) Visual representations are fused with non-visual geospatial context from the Population Dynamics Foundation Model (PDFM) agarwal2024general. The final representations are then used to predict walkability scores.
• Figure 2: Overview of the Spatially-Aware Feature Enhancement (SAFE) module. Each location aggregates visual representations from nearby neighbors with distance-based weights. The combined representation is passed through a multilayer perceptron (MLP) to predict walkability scores.
• Figure 3: Walk Score distribution across Minneapolis–Saint Paul. Each cell represents a neighborhood-level area colored by its Walk Score value. These scores reflect the walkability of the built environment based on factors such as proximity to amenities, street connectivity, and pedestrian infrastructure. Higher scores (yellow) indicate highly walkable areas, while lower scores (dark blue) correspond to less walkable regions.
• Figure 4: Attention heatmaps from WalkCLIP highlighting visual cues associated with walkability. (a) In satellite imagery, the model focuses on residential street layouts, sidewalks, and community amenities such as parks, playgrounds, and recreational facilities. (b) In street view imagery, high attension score are on sidewalks, bike lanes, and tree cover.
• Figure 5: Case study examples showing paired satellite (left) and street view (right) imagery.