PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models

TL;DR

PlanGPT-VL introduces a domain-specific Vision-Language Model tailored for urban planning map interpretation, addressing hallucinations and domain-language alignment with a three-part framework: PlanAnno-V data synthesis, Critical Point Thinking verification, and PlanBench-V domain-specific evaluation. It curates a high-quality, expert-annotated training pipeline from thousands of planning maps, expands instruction coverage, and employs a Generate-Verify-Revise loop to reduce errors. The model fine-tunes a compact 7B parameter backbone while freezing the vision encoder, achieving superior domain performance and competitive general capabilities, and surpassing general VLMs by substantial margins on planning tasks. The PlanBench-V benchmark, along with comprehensive ablations, demonstrates the value of domain specialization for planning tasks and offers a blueprint for extending domain-focused VLMs to other specialized modalities and regions.

Abstract

In the field of urban planning, existing Vision-Language Models (VLMs) frequently fail to effectively analyze and evaluate planning maps, despite the critical importance of these visual elements for urban planners and related educational contexts. Planning maps, which visualize land use, infrastructure layouts, and functional zoning, require specialized understanding of spatial configurations, regulatory requirements, and multi-scale analysis. To address this challenge, we introduce PlanGPT-VL, the first domain-specific Vision-Language Model tailored specifically for urban planning maps. PlanGPT-VL employs three innovative approaches: (1) PlanAnno-V framework for high-quality VQA data synthesis, (2) Critical Point Thinking to reduce hallucinations through structured verification, and (3) comprehensive training methodology combining Supervised Fine-Tuning with frozen vision encoder parameters. Through systematic evaluation on our proposed PlanBench-V benchmark, we demonstrate that PlanGPT-VL significantly outperforms general-purpose state-of-the-art VLMs in specialized planning map interpretation tasks, offering urban planning professionals a reliable tool for map analysis, assessment, and educational applications while maintaining high factual accuracy. Our lightweight 7B parameter model achieves comparable performance to models exceeding 72B parameters, demonstrating efficient domain specialization without sacrificing performance.

Figures (14)

• Figure 1: Urban planning multimodal tasks including map elements identification, spatial relationships understanding, expert reasoning, policy association and other key applications.
• Figure 2: Overview of PlanAnno-V framework. Our approach synthesizes high-quality instruction-response pairs through a three-stage process: (1) domain-specific data preprocessing with expert annotation, (2) instruction-response synthesis using Critical Point Thinking for hallucination reduction, and (3) model-specific rewriting to align with professional planning communication patterns.
• Figure 3: Analysis of PlanAnno-V instruction synthesis: (a) UMAP projection of instruction embeddings with kernel density estimation contours, showing how synthesized instructions (blue) maintain similar distribution patterns to expert-annotated seed data (red) while introducing beneficial diversity; (b) Categorical distribution of synthesized instructions across planning dimensions; (c) Statistical distribution of PlanBench-V Dataset.
• Figure 4: Attention visualization comparing models with and without caption integration.
• Figure 5: Token Distribution and Cirtical Point Distribution Analysis