Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models

Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li

TL;DR

UrbanPlanBench introduces a dedicated benchmark and UrbanPlanText dataset to quantify large language models' proficiency in urban planning, a domain where expert human judgment remains essential. The benchmark evaluates three core perspectives—fundamental principles, professional knowledge, and management/regulations—via MCQ-based tasks modeled after real China urban planner exams, revealing substantial gaps between current LLMs and human professionals. The study demonstrates that prompting techniques (RAG, Chain-of-Thought) and large-scale model scaling improve performance, but MCQ-M items remain particularly challenging, and language and subject biases influence results. By publicly releasing the benchmark, dataset, and tooling, the work aims to catalyze practical AI-assisted urban planning while identifying clear directions for future improvements, including multilingual and multimodal extensions.

Abstract

The advent of Large Language Models (LLMs) holds promise for revolutionizing various fields traditionally dominated by human expertise. Urban planning, a professional discipline that fundamentally shapes our daily surroundings, is one such field heavily relying on multifaceted domain knowledge and experience of human experts. The extent to which LLMs can assist human practitioners in urban planning remains largely unexplored. In this paper, we introduce a comprehensive benchmark, UrbanPlanBench, tailored to evaluate the efficacy of LLMs in urban planning, which encompasses fundamental principles, professional knowledge, and management and regulations, aligning closely with the qualifications expected of human planners. Through extensive evaluation, we reveal a significant imbalance in the acquisition of planning knowledge among LLMs, with even the most proficient models falling short of meeting professional standards. For instance, we observe that 70% of LLMs achieve subpar performance in understanding planning regulations compared to other aspects. Besides the benchmark, we present the largest-ever supervised fine-tuning (SFT) dataset, UrbanPlanText, comprising over 30,000 instruction pairs sourced from urban planning exams and textbooks. Our findings demonstrate that fine-tuned models exhibit enhanced performance in memorization tests and comprehension of urban planning knowledge, while there exists significant room for improvement, particularly in tasks requiring domain-specific terminology and reasoning. By making our benchmark, dataset, and associated evaluation and fine-tuning toolsets publicly available at https://github.com/tsinghua-fib-lab/PlanBench, we aim to catalyze the integration of LLMs into practical urban planning, fostering a symbiotic collaboration between human expertise and machine intelligence.

UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models

TL;DR

UrbanPlanBench introduces a dedicated benchmark and UrbanPlanText dataset to quantify large language models' proficiency in urban planning, a domain where expert human judgment remains essential. The benchmark evaluates three core perspectives—fundamental principles, professional knowledge, and management/regulations—via MCQ-based tasks modeled after real China urban planner exams, revealing substantial gaps between current LLMs and human professionals. The study demonstrates that prompting techniques (RAG, Chain-of-Thought) and large-scale model scaling improve performance, but MCQ-M items remain particularly challenging, and language and subject biases influence results. By publicly releasing the benchmark, dataset, and tooling, the work aims to catalyze practical AI-assisted urban planning while identifying clear directions for future improvements, including multilingual and multimodal extensions.

Abstract

The advent of Large Language Models (LLMs) holds promise for revolutionizing various fields traditionally dominated by human expertise. Urban planning, a professional discipline that fundamentally shapes our daily surroundings, is one such field heavily relying on multifaceted domain knowledge and experience of human experts. The extent to which LLMs can assist human practitioners in urban planning remains largely unexplored. In this paper, we introduce a comprehensive benchmark, UrbanPlanBench, tailored to evaluate the efficacy of LLMs in urban planning, which encompasses fundamental principles, professional knowledge, and management and regulations, aligning closely with the qualifications expected of human planners. Through extensive evaluation, we reveal a significant imbalance in the acquisition of planning knowledge among LLMs, with even the most proficient models falling short of meeting professional standards. For instance, we observe that 70% of LLMs achieve subpar performance in understanding planning regulations compared to other aspects. Besides the benchmark, we present the largest-ever supervised fine-tuning (SFT) dataset, UrbanPlanText, comprising over 30,000 instruction pairs sourced from urban planning exams and textbooks. Our findings demonstrate that fine-tuned models exhibit enhanced performance in memorization tests and comprehension of urban planning knowledge, while there exists significant room for improvement, particularly in tasks requiring domain-specific terminology and reasoning. By making our benchmark, dataset, and associated evaluation and fine-tuning toolsets publicly available at https://github.com/tsinghua-fib-lab/PlanBench, we aim to catalyze the integration of LLMs into practical urban planning, fostering a symbiotic collaboration between human expertise and machine intelligence.
Paper Structure (18 sections, 4 figures, 5 tables)

This paper contains 18 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Benchmarking LLMs on Urban Planning
  3. Construction of UrbanPlanBench
  4. Evaluation Results
  5. Overall planning capabilities
  6. Subject imbalance
  7. Language bias
  8. Scaling effect
  9. Longitudinal studies
  10. Prompting Techniques
  11. Fine-tuning LLMs with UrbanPlanText
  12. Dataset Construction
  13. SFT Data Quality
  14. SFT Results
  15. Related Work
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Example questions of UrbanPlanBench. MCQ-S has four options where only one option is correct. MCQ-M is more challenging, featuring two to four correct options from a total of five options. Contents are translated from Chinese to English.
  • Figure 2: Performance of different model sizes. The LLM model Qwen1.5 is adopted. MCS-S and MCQ-M indicate MCQs with one single correct answer and multiple correct answers, respectively. (Left) Accuracy on MCQ-S questions. (Right) Accuracy on MCQ-M questions.
  • Figure 3: Data collection and process of UrbanPlanText. Past exam questions are first annotated by the authors into structured CSV files and then transformed into MCQ-type instruction pairs and dialog-style instruction pairs. Textbooks are first parsed into textual files, from which instruction pairs are generated automatically by prompting OpenAI's ChatGPT model. Contents are translated from Chinese to English.
  • Figure 4: Performance of different model sizes after SFT on UrbanPlanText. The LLM model Qwen1.5 is adopted. MCS-S and MCQ-M indicate MCQs with one single correct answer and multiple correct answers, respectively. (Left) Accuracy on MCQ-S questions. (Right) Accuracy on MCQ-M questions.