Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models

Li-Zhong Szu-Tu, Ting-Lin Wu, Chia-Jui Chang, He Syu, Yu-Lun Liu

TL;DR

This work introduces YearGuessr, the first global, open benchmark for building-age estimation with continuous ordinal labels from 55k facade images across 157 countries, enriched with GPS and popularity signals. It then proposes YearCLIP, a CLIP-based model that fuses visual features with location priors and reasoning prompts to perform coarse-to-fine ordinal regression, plus an explainability mechanism. Through a comprehensive benchmark of 30+ models, the study reveals a pronounced popularity (memorization) bias in vision-language models, where famous landmarks yield dramatically higher accuracy than lesser-known buildings, and shows that ordinal training and geo-aware cues can mitigate some biases. The results highlight the need for robust, regionally balanced datasets and bias-conscious evaluation protocols to ensure reliable architectural dating and geo-aware reasoning. The work provides a practical resource for heritage, retrofit planning, and disaster assessment, while also cautioning about potential biases and the responsible use of the data.

Abstract

We expose a significant popularity bias in state-of-the-art vision-language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open benchmark for this task: the YearGuessr dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/

Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models

TL;DR

This work introduces YearGuessr, the first global, open benchmark for building-age estimation with continuous ordinal labels from 55k facade images across 157 countries, enriched with GPS and popularity signals. It then proposes YearCLIP, a CLIP-based model that fuses visual features with location priors and reasoning prompts to perform coarse-to-fine ordinal regression, plus an explainability mechanism. Through a comprehensive benchmark of 30+ models, the study reveals a pronounced popularity (memorization) bias in vision-language models, where famous landmarks yield dramatically higher accuracy than lesser-known buildings, and shows that ordinal training and geo-aware cues can mitigate some biases. The results highlight the need for robust, regionally balanced datasets and bias-conscious evaluation protocols to ensure reliable architectural dating and geo-aware reasoning. The work provides a practical resource for heritage, retrofit planning, and disaster assessment, while also cautioning about potential biases and the responsible use of the data.

Abstract

We expose a significant popularity bias in state-of-the-art vision-language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open benchmark for this task: the YearGuessr dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/
Paper Structure (61 sections, 8 figures, 8 tables)

This paper contains 61 sections, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets for building-age estimation.
  4. Image geolocalization foundations.
  5. Geo-aware VLMs.
  6. Ordinal regression with numeric cues.
  7. Multi-modal learning signals.
  8. Dataset and Benchmark
  9. Dataset Construction
  10. Data sources and licences.
  11. Automatic collection pipeline.
  12. Cleaning and quality control.
  13. Train/validation/test split.
  14. Metadata completeness.
  15. Statistics Analysis
  16. ...and 46 more sections

Figures (8)

  • Figure 1: Data collection and cleaning pipeline. (a) We crawl the Wikipedia category tree of buildings, collecting façade images, construction years, GPS coordinates, textual descriptions, and pageview statistics. (b) The raw crawl of 90k images is refined through deduplication, a CLIP-based building filter, and a light manual audit, yielding 55k clean façades. (c) Examples of discarded non-building or duplicate samples.
  • Figure 2: Dataset statistics. This figure provides an overview of our dataset's key characteristics. (a) Continent Distribution shows the geographical origins of the images. (b) Built Year Distribution illustrates the age of the structures. (c) Pageview Distribution represents the buildings' popularity. (d) Renovation Distribution indicates the extent of reconstruction. (e) Rural/Urban Distribution reflects the population density of a building's location.
  • Figure 3: YearCLIP architecture. An image encoder $f_v$ (CLIP) extracts 224$\times$224 facade features. We then fuse the feature with a GPS embedding from the location encoder $f_l$ (RFF + MLP, optional input) via a learnable zero-convolution. Parallel text branches encode (i) seven coarse style classes $f_c$ and (ii) a bank of reasoning prompts $f_r$ describing roofs, walls, heights, etc. All frozen encoders feed a trainable regressor $g(\cdot)$ that performs coarse-to-fine ordinal regression. It predicts a construction year (here 1687), selects the best-matching style/reason tokens, and outputs a readable rationale.
  • Figure 4: Prediction error scatter plots for representative models. (a) ConvNeXt-B (CNN), (b) Swin-B (Transformer), (c) YearCLIP (ours, CLIP-based), (d) Gemini1.5-pro (VLM), and (e) Gemma3-27B (VLM). The horizontal axis shows predicted construction year, vertical axis shows groundtruth. Each point represents a single building. The red diagonal line indicates perfect prediction.
  • Figure 5: Explainable age predictions with YearCLIP. Powered by Reason-enhanced NumCLIP, the system predicts construction year within ±15 yr of ground truth and provides rationales that highlight stylistic and historic cues. CLIP baselines miss or misassign these signals, whereas our location + reason pipeline yields transparent, verifiable explanations.
  • ...and 3 more figures