Low-Pass Filtering Improves Behavioral Alignment of Vision Models

TL;DR

It is shown that the frequency spectrum of optimal Gaussian filters roughly matches the spectrum of band-pass filters implemented by the human visual system, and the contrast sensitivity function is approximated well by Gaussian filters of the specific width that also maximizes error consistency.

Abstract

Despite their impressive performance on computer vision benchmarks, Deep Neural Networks (DNNs) still fall short of adequately modeling human visual behavior, as measured by error consistency and shape bias. Recent work hypothesized that behavioral alignment can be drastically improved through \emph{generative} -- rather than \emph{discriminative} -- classifiers, with far-reaching implications for models of human vision. Here, we instead show that the increased alignment of generative models can be largely explained by a seemingly innocuous resizing operation in the generative model which effectively acts as a low-pass filter. In a series of controlled experiments, we show that removing high-frequency spatial information from discriminative models like CLIP drastically increases their behavioral alignment. Simply blurring images at test-time -- rather than training on blurred images -- achieves a new state-of-the-art score on the model-vs-human benchmark, halving the current alignment gap between DNNs and human observers. Furthermore, low-pass filters are likely optimal, which we demonstrate by directly optimizing filters for alignment. To contextualize the performance of optimal filters, we compute the frontier of all possible pareto-optimal solutions to the benchmark, which was formerly unknown. We explain our findings by observing that the frequency spectrum of optimal Gaussian filters roughly matches the spectrum of band-pass filters implemented by the human visual system. We show that the contrast sensitivity function, describing the inverse of the contrast threshold required for humans to detect a sinusoidal grating as a function of spatiotemporal frequency, is approximated well by Gaussian filters of the specific width that also maximizes error consistency.

Figures (14)

• Figure 1: Low-pass filtering images increases human-machine Error Consistency. For the majority of investigated models, we find that low-pass filtering images prior to model evaluation can substantially increase their error consistency with human observers.
• Figure 2: Low-pass filtering test stimuli improves behavioral alignment. Gaussian blurring [A+C] and resizing [B+D] both lead to higher shape bias (A+B) and error consistency (C+D). While the shape bias strictly increases under either transformation, the error consistency reaches a maximum "critical point" and declines afterwards.
• Figure 3: The optimal filter is a low-pass filter. We plot the log-transformed, center-shifted amplitudes of the optimal Fourier filter, as a function of the horizontal and vertical frequencies in cycles per image.
• Figure 4: Low-pass filters approximate the CSF well. Left: Contrast sensitivity at a presentation time of 200ms as a function of spatial frequency (red curve) is approximated well by the best Gaussian filter (blue curve). Right: The best-fitting Gaussian has a $\sigma$ of about $2.5px$, matching our empirical result. Evidently, this finding is robust to the exact choice of $\beta$.
• Figure 5: Removing high-frequency information from test stimuli improves behavioral alignment for a wide range of models. We measure shape bias and error consistency for ResNet; SWSL, BiT-M, ViT, Noisy Student, and a variety of OpenCLIP models. We find that in general, shape bias [A+B] increases with blur and resize strength, while error consistency [C+D] usually increases at first before dropping off again.