Text Chunking for Document Classification for Urban System Management using Large Language Models

TL;DR

Qualitative coding of urban-system documentation is resource-intensive and prone to human variability. The study compares whole-text and 500-word chunk prompting across three LLMs (GPT-4o, GPT-4o-mini, o1-mini) to deductively code 17 digital twin characteristics in 10 papers, using manual coding as ground truth. Findings show that chunking generally improves internal consistency and agreement with human raters, with GPT-4o and o1-mini achieving strong recall and significant consensus when used with chunked prompts. The work demonstrates a scalable, reproducible AI-assisted coding workflow for urban systems management that can inform policy and infrastructure assessments.

Abstract

Urban systems are managed using complex textual documentation that need coding and analysis to set requirements and evaluate built environment performance. This paper contributes to the study of applying large-language models (LLM) to qualitative coding activities to reduce resource requirements while maintaining comparable reliability to humans. Qualitative coding and assessment face challenges like resource limitations and bias, accuracy, and consistency between human evaluators. Here we report the application of LLMs to deductively code 10 case documents on the presence of 17 digital twin characteristics for the management of urban systems. We utilize two prompting methods to compare the semantic processing of LLMs with human coding efforts: whole text analysis and text chunk analysis using OpenAI's GPT-4o, GPT-4o-mini, and o1-mini models. We found similar trends of internal variability between methods and results indicate that LLMs may perform on par with human coders when initialized with specific deductive coding contexts. GPT-4o, o1-mini and GPT-4o-mini showed significant agreement with human raters when employed using a chunking method. The application of both GPT-4o and GPT-4o-mini as an additional rater with three manual raters showed statistically significant agreement across all raters, indicating that the analysis of textual documents is benefited by LLMs. Our findings reveal nuanced sub-themes of LLM application suggesting LLMs follow human memory coding processes where whole-text analysis may introduce multiple meanings. The novel contributions of this paper lie in assessing the performance of OpenAI GPT models and introduces the chunk-based prompting approach, which addresses context aggregation biases by preserving localized context.

Figures (6)

• Figure 1: Internal Agreement of LLMs and Prompting Approach. The blue line shows the average internal agreement across all papers while the grey lines show the individual results of each paper.
• Figure 2: Averaged LLM Accuracy with Manual Raters with $\pm 2$ Standard Deviations of Error.
• Figure 3: Precision vs. Recall of Models across all classifications using consensus approach when compared to manual consensus results across all 10 analyzed documents.
• Figure 4: Confusion Matrix results of the different approaches when comparing the consensus of 15 iterations of LLM execution to the consensus of 3 manual raters.
• Figure 5: Consensus Approach Comparison to human raters for each dimension across all 10 papers, showing the true positive and true negative classifications. The background white bar shows the number of positive or negative classifications by the manual consensus.