How to Build Robust, Scalable Models for GSV-Based Indicators in Neighborhood Research

TL;DR

This work tackles the domain shift from ImageNet-pretrained vision models to Google Street View imagery for neighborhood-health indicators. It systematically compares Vision Transformer (ViT) and Vision Mamba (Vim) backbones, applying unsupervised post-training with DINO on 1M unlabeled GSV images and evaluating via fine-tuning and linear probing on multiple downstream tasks. Key findings show that Vim-S after post-training offers strong downstream generalization, but gains depend on architecture, data curation, and training regime; linear probing can reveal domain mismatch and representation drift, underscoring the importance of data quality and careful evaluation. The study provides practical guidance for researchers in neighborhood research on model choice, SSL strategies, and data handling to build robust, scalable GSV-based indicators.

Abstract

A substantial body of health research demonstrates a strong link between neighborhood environments and health outcomes. Recently, there has been increasing interest in leveraging advances in computer vision to enable large-scale, systematic characterization of neighborhood built environments. However, the generalizability of vision models across fundamentally different domains remains uncertain, for example, transferring knowledge from ImageNet to the distinct visual characteristics of Google Street View (GSV) imagery. In applied fields such as social health research, several critical questions arise: which models are most appropriate, whether to adopt unsupervised training strategies, what training scale is feasible under computational constraints, and how much such strategies benefit downstream performance. These decisions are often costly and require specialized expertise. In this paper, we answer these questions through empirical analysis and provide practical insights into how to select and adapt foundation models for datasets with limited size and labels, while leveraging larger, unlabeled datasets through unsupervised training. Our study includes comprehensive quantitative and visual analyses comparing model performance before and after unsupervised adaptation.

Figures (9)

• Figure 1: Random GSV image samples used in unsupervised training. Original image size: 640 x 440.
• Figure 2: Model Architectures(Left: ViT; Right: Vim). The gray rectangles labeled from 0 to N represent the positional embeddings.
• Figure 3: Model Architectures(Left: ViT; Right: Vim). The gray rectangles labeled from 0 to N represent the positional embeddings.
• Figure 4: Comparison of token dependencies(Lines with arrows) and receptive field(Black boxes) for different architectures.
• Figure 5: Illustration of our post-training and evaluation pipeline. The backbone network can be instantiated with any architecture; here we use ViT and Vim. During post-training, model weights are updated(indicated by “fire”). During inference, we either fine-tune or freeze the pretrained backbone(indicated by “freeze”) while updating the classification head.