VFA: Vision Frequency Analysis of Foundation Models and Human

TL;DR

The paper investigates how characteristics of large-scale vision models align with human perception and robustness under distribution shifts. It introduces a frequency-band analysis that masks bands and quantifies bandwidth as the width at half-maximum on a logarithmic scale across over a thousand models, including CLIP- and BEiT-based variants. Key findings show that increasing model size and data (notably ImageNet-22K) reduces bandwidth toward human levels, and semantic tokenization with CLIP supervision further aligns bandwidth, with bandwidth serving as a strong predictor of OOD accuracy. Practically, these results suggest scaling and multimodal guidance as viable paths to more robust, human-aligned vision systems and inform future benchmarks and brain-inspired research.

Abstract

Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception may achieve higher out-of-distribution generalization. In this study, we investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness. Our findings indicate that increasing model and data size and incorporating rich semantic information and multiple modalities enhance models' alignment with human perception and their overall robustness. Our empirical analysis demonstrates a strong correlation between out-of-distribution accuracy and human alignment.

Figures (7)

• Figure 1: Frequency Bandwidths of Humans and Models. Humans are sensitive to a narrow frequency band, and adding noise within this band (under the red curve) degrades their performance. In contrast, models exhibit a wider frequency band (green curves), making them more vulnerable to noise across a broader range of frequencies. Narrowing the band might improve robustness.
• Figure 2: Correlation between Bandwidth (BW) and Model Size in Logarithmic Scale. The regression line represents that as the model size scales, the bandwidth decreases, converging towards human levels. Each dot corresponds to a model, for model names and details, see Section \ref{['sec:Scaling_Experiment_details']} in the Appendix.
• Figure 3: Effect of Data on Bandwidth: Comparing models trained with ImageNet-22K. Models that benefit from ImageNet-22K training exhibit significantly smaller bandwidths.
• Figure 4: BW and Shape bias comparison. A) OOD versus Bandwidth. B) OOD versus Shape bias. Both BW and Shape bias are predictive of OOD performance and are correlated, with BW showing a higher correlation with OOD generalization.
• Figure 5: Case studies. (A) Comparing Bandwidth of BEiT Models. (B) Human-like models. Many models exhibit human-like bandwidth, with a version of the BEiT-V2 model perfectly matching the human curve.