CityGPT: Empowering Urban Spatial Cognition of Large Language Models

TL;DR

CityGPT presents a city-focused, instruction-tuned framework to endow large language models with robust urban spatial cognition. By constructing CityInstruction from embodied mobility data, applying self-weighted fine-tuning (SWFT) to preserve general capabilities, and evaluating through the CityEval benchmark, the approach enables smaller LLMs to achieve competitive performance with proprietary systems across multiple cities. Key contributions include CityInstruction, SWFT, and CityEval, plus extensive ablations showing the value of multi-source data and cross-city transferability. The framework promises practical impact for urban planning, navigation, and geospatial reasoning by grounding language models in realistic, city-scale spatial knowledge.

Abstract

Large language models(LLMs), with their powerful language generation and reasoning capabilities, have already achieved notable success in many domains, e.g., math and code generation. However, they often fall short when tackling real-life geospatial tasks within urban environments. This limitation stems from a lack of physical world knowledge and relevant data during training. To address this gap, we propose \textit{CityGPT}, a systematic framework designed to enhance LLMs' understanding of urban space and improve their ability to solve the related urban tasks by integrating a city-scale `world model' into the model. Firstly, we construct a diverse instruction tuning dataset, \textit{CityInstruction}, for injecting urban knowledge into LLMs and effectively boosting their spatial reasoning capabilities. Using a combination of \textit{CityInstruction} and open source general instruction data, we introduce a novel and easy-to-use self-weighted fine-tuning method (\textit{SWFT}) to train various LLMs (including ChatGLM3-6B, Llama3-8B, and Qwen2.5-7B) to enhance their urban spatial capabilities without compromising, or even improving, their general abilities. Finally, to validate the effectiveness of our proposed framework, we develop a comprehensive text-based spatial benchmark \textit{CityEval} for evaluating the performance of LLMs across a wide range of urban scenarios and geospatial tasks. Extensive evaluation results demonstrate that smaller LLMs trained with \textit{CityInstruction} by \textit{SWFT} method can achieve performance that is competitive with, and in some cases superior to, proprietary LLMs when assessed using \textit{CityEval}.

Figures (9)

• Figure 1: An overview of CityGPT, including CityInstruction dataset, self-weighted tuning SWFT method, and CityEval benchmark. CityInstruction comprises CityQA, CityWalk and CityReasoning, while CityEval includes City Image, Urban Semantics, Spatial Reasoning and Composite Tasks.
• Figure 2: The loss of data samples before and after training, where the dashed trend line represents the average learning ratio, and red pentagrams highlight the anomalous region.
• Figure 3: Composition of CityEval.
• Figure 4: The performance of CityGPT@Beijing consistently exceeds that of the baseline across various base models on CityEval benchmark.
• Figure 5: Effectiveness of proposed self-weighted tuning.