StreetLens: Enabling Human-Centered AI Agents for Neighborhood Assessment from Street View Imagery
Jina Kim, Leeje Jang, Yao-Yi Chiang, Guanyu Wang, Michelle C. Pasco
TL;DR
StreetLens addresses the scalability gap in neighborhood environmental assessment by embedding domain knowledge into a configurable, end-to-end workflow that uses vision-language models to emulate trained human coders. It comprises four modules—data processing, automated prompt tuning, VLM processing, and feedback provision—to retrieve street view imagery, align prompts with coding protocols, generate structured annotations, and provide interpretable explanations. The approach leverages domain-informed prompts, codebooks, and in-context learning with training materials to enable cross-context applicability across geographic settings and prior survey data integration. By providing an accessible Google Colab workflow and open-source components, StreetLens aims to accelerate robust neighborhood studies while maintaining interpretability and researcher control.
Abstract
Traditionally, neighborhood studies have used interviews, surveys, and manual image annotation guided by detailed protocols to identify environmental characteristics, including physical disorder, decay, street safety, and sociocultural symbols, and to examine their impact on developmental and health outcomes. Although these methods yield rich insights, they are time-consuming and require intensive expert intervention. Recent technological advances, including vision language models (VLMs), have begun to automate parts of this process; however, existing efforts are often ad hoc and lack adaptability across research designs and geographic contexts. In this paper, we present StreetLens, a user-configurable human-centered workflow that integrates relevant social science expertise into a VLM for scalable neighborhood environmental assessments. StreetLens mimics the process of trained human coders by focusing the analysis on questions derived from established interview protocols, retrieving relevant street view imagery (SVI), and generating a wide spectrum of semantic annotations from objective features (e.g., the number of cars) to subjective perceptions (e.g., the sense of disorder in an image). By enabling researchers to define the VLM's role through domain-informed prompting, StreetLens places domain knowledge at the core of the analysis process. It also supports the integration of prior survey data to enhance robustness and expand the range of characteristics assessed in diverse settings. StreetLens represents a shift toward flexible and agentic AI systems that work closely with researchers to accelerate and scale neighborhood studies. StreetLens is publicly available at https://knowledge-computing.github.io/projects/streetlens.