Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery

Zhenyuan Yang, Xuhui Lin, Qinyi He, Ziye Huang, Zhengliang Liu, Hanqi Jiang, Peng Shu, Zihao Wu, Yiwei Li, Stephen Law, Gengchen Mai, Tianming Liu, Tao Yang

TL;DR

This work systematically evaluates ChatGPT-4V and Gemini Pro across Street View Imagery, Built Environment, and Interior tasks to characterize multimodal foundation models in urban contexts. It demonstrates strengths in scale-oriented reasoning (e.g., road widths, building heights) and holistic descriptions (interior style and room classification) while revealing weaknesses in fine-grained counting and precise measurements under occlusion or complex scenes. The study highlights zero-shot capabilities but also domain-dependent variability, underscoring the need for domain-specific fine-tuning, diverse and privacy-conscious training data, and careful integration with traditional urban planning tools. Overall, the findings illustrate foundational multimodal intelligence with significant practical potential for urban studies, planning, and design, provided ethical and methodological considerations are addressed.

Abstract

The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture identification, pedestrian and car counts, and road width measurement in Street View Imagery; building function classification, building age analysis, building height analysis, and building structure classification in the Built Environment; and interior room classification, interior design style analysis, interior furniture counts, and interior length measurement in Interior. The results reveal proficiency in length measurement, style analysis, question answering, and basic image understanding, but highlight limitations in detailed recognition and counting tasks. While zero-shot learning shows potential, performance varies depending on the problem domains and image complexities. This study provides new insights into the strengths and weaknesses of multimodal foundation models for practical challenges in Street View Imagery, Built Environment, and Interior. Overall, the findings demonstrate foundational multimodal intelligence, emphasizing the potential of FMs to drive forward interdisciplinary applications at the intersection of computer vision and language.

Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery

TL;DR

This work systematically evaluates ChatGPT-4V and Gemini Pro across Street View Imagery, Built Environment, and Interior tasks to characterize multimodal foundation models in urban contexts. It demonstrates strengths in scale-oriented reasoning (e.g., road widths, building heights) and holistic descriptions (interior style and room classification) while revealing weaknesses in fine-grained counting and precise measurements under occlusion or complex scenes. The study highlights zero-shot capabilities but also domain-dependent variability, underscoring the need for domain-specific fine-tuning, diverse and privacy-conscious training data, and careful integration with traditional urban planning tools. Overall, the findings illustrate foundational multimodal intelligence with significant practical potential for urban studies, planning, and design, provided ethical and methodological considerations are addressed.

Abstract

The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture identification, pedestrian and car counts, and road width measurement in Street View Imagery; building function classification, building age analysis, building height analysis, and building structure classification in the Built Environment; and interior room classification, interior design style analysis, interior furniture counts, and interior length measurement in Interior. The results reveal proficiency in length measurement, style analysis, question answering, and basic image understanding, but highlight limitations in detailed recognition and counting tasks. While zero-shot learning shows potential, performance varies depending on the problem domains and image complexities. This study provides new insights into the strengths and weaknesses of multimodal foundation models for practical challenges in Street View Imagery, Built Environment, and Interior. Overall, the findings demonstrate foundational multimodal intelligence, emphasizing the potential of FMs to drive forward interdisciplinary applications at the intersection of computer vision and language.
Paper Structure (59 sections, 126 figures)

This paper contains 59 sections, 126 figures.

Table of Contents

  1. Introduction
  2. Background
  3. The Rise of LLMs and Multimodal Models
  4. Street View Imagery
  5. Built Environment
  6. Interior
  7. Experiments and Observation of FMs for Street Furniture
  8. Brand Recognition
  9. Data Source
  10. Evaluation and analysis
  11. Counts of Pedestrians
  12. Data Source
  13. Evaluation and analysis
  14. Counts of Cars
  15. Data Source
  16. ...and 44 more sections

Figures (126)

  • Figure 1: Brand Recognition of Burger King in GPT-4V
  • Figure 2: Brand Recognition of Apple in GPT-4V
  • Figure 3: Brand Recognition of Starbucks in GPT-4V
  • Figure 4: Brand Recognition of KFC in GPT-4V
  • Figure 5: Brand Recognition of Burger King in GPT-4o
  • ...and 121 more figures