Decoding Neighborhood Environments with Large Language Models

TL;DR

This paper tackles scalable decoding of neighborhood environments from street-view imagery by combining a strong supervised detector (YOLOv11 Nano) with an evaluation of multiple large language models to identify environmental indicators without additional training. It demonstrates that YOLOv11 achieves an average mAP50 of 99.1% and F1 of 0.963 across six indicators, while LLMs—when prompted effectively and aggregated via majority voting—approach 88% overall accuracy. The work systematically analyzes prompting strategies, multilingual robustness, and parameter tuning, identifying practical challenges such as language bias and computational costs of ensemble approaches. The findings suggest LLM-based, training-free decoding of neighborhood environments is feasible at scale and could significantly reduce labeling and labeling-related resource demands for large geographic areas.

Abstract

Neighborhood environments include physical and environmental conditions such as housing quality, roads, and sidewalks, which significantly influence human health and well-being. Traditional methods for assessing these environments, including field surveys and geographic information systems (GIS), are resource-intensive and challenging to evaluate neighborhood environments at scale. Although machine learning offers potential for automated analysis, the laborious process of labeling training data and the lack of accessible models hinder scalability. This study explores the feasibility of large language models (LLMs) such as ChatGPT and Gemini as tools for decoding neighborhood environments (e.g., sidewalk and powerline) at scale. We train a robust YOLOv11-based model, which achieves an average accuracy of 99.13% in detecting six environmental indicators, including streetlight, sidewalk, powerline, apartment, single-lane road, and multilane road. We then evaluate four LLMs, including ChatGPT, Gemini, Claude, and Grok, to assess their feasibility, robustness, and limitations in identifying these indicators, with a focus on the impact of prompting strategies and fine-tuning. We apply majority voting with the top three LLMs to achieve over 88% accuracy, which demonstrates LLMs could be a useful tool to decode the neighborhood environment without any training effort.

Figures (8)

• Figure 1: Overview of Methodology.
• Figure 2: Accuracy with augmentation
• Figure 3: Impact of different SNR levels
• Figure 4: Accuracy of LLMs in parallel and sequence prompts.
• Figure 5: Accuracy of LLMs and Majority Voting.