Can a Large Language Model Assess Urban Design Quality? Evaluating Walkability Metrics Across Expertise Levels

TL;DR

This study investigates how embedding formal urban-design expertise in prompts can improve a large language model's ability to assess walkability from street view imagery. By compiling 124 literature-based walkability metrics and structuring them with a Triple-A ontology, the authors compare four prompting configurations (C1–C4) across 42 Singapore SVIs to evaluate safety and attractiveness. Results show that LLMs can perform assessments from general knowledge, but expert-knowledge prompts yield more consistent, concentration-rich scoring and reduce misinterpretations, with C1 tending to overestimate. The work highlights both the potential and the risks of automated multimodal walkability evaluation, underscoring the need for standardized prompt design and a formalized walkability ontology to support scalable urban-design decision support.

Abstract

Urban street environments are vital to supporting human activity in public spaces. The emergence of big data, such as street view images (SVIs) combined with multimodal large language models (MLLMs), is transforming how researchers and practitioners investigate, measure, and evaluate semantic and visual elements of urban environments. Considering the low threshold for creating automated evaluative workflows using MLLMs, it is crucial to explore both the risks and opportunities associated with these probabilistic models. In particular, the extent to which the integration of expert knowledge can influence the performance of MLLMs in evaluating the quality of urban design has not been fully explored. This study sets out an initial exploration of how integrating more formal and structured representations of expert urban design knowledge into the input prompts of an MLLM (ChatGPT-4) can enhance the model's capability and reliability in evaluating the walkability of built environments using SVIs. We collect walkability metrics from the existing literature and categorize them using relevant ontologies. We then select a subset of these metrics, focusing on the subthemes of pedestrian safety and attractiveness, and develop prompts for the MLLM accordingly. We analyze the MLLM's ability to evaluate SVI walkability subthemes through prompts with varying levels of clarity and specificity regarding evaluation criteria. Our experiments demonstrate that MLLMs are capable of providing assessments and interpretations based on general knowledge and can support the automation of multimodal image-text evaluations. However, they generally provide more optimistic scores and can make mistakes when interpreting the provided metrics, resulting in incorrect evaluations. By integrating expert knowledge, the MLLM's evaluative performance exhibits higher consistency and concentration.

Figures (7)

• Figure 1: Streets in Singapore were selected for evaluating SVI walkability, including streets from the downtown area, countryside, commercial centre, and residential areas. (c) KartaView contributors
• Figure 2: Plot of the streets' average scores in safety and attractiveness assessments, as evaluated by the four MLLMs.
• Figure 3: Street view images with the highest and lowest scores for safety (top) and attractiveness (bottom), giving the corresponding scores.
• Figure 4: The score distributions of the four MLLMs assessing safety (top) and attractiveness (bottom).
• Figure 5: The top six metrics with the largest statistical differences according to safety scores (top) and attractiveness scores (bottom), ranked by their test statistics. The density plots show the level of concentration in each MLLM when measuring the particular metric.