Are Convolutional Neural Networks or Transformers more like human vision?

TL;DR

The paper investigates whether Vision Transformers (ViT) exhibit more human-like vision than CNNs by analyzing error patterns beyond accuracy. It introduces error-consistency metrics (observed error overlap, Cohen's $\kappa$, class-wise and inter-class Jensen-Shannon distances) and applies them to a diagnostic Stylized ImageNet dataset to compare CNNs and ViTs. Results show ViT yields higher human-aligned error consistency across several metrics, though inter-class patterns reveal nuanced differences, and shape bias partially explains some findings. The work demonstrates that beyond achieving higher accuracy, transformers may implement decision strategies more akin to human vision, and reveals how data augmentation can shift biases toward shape while impacting performance.

Abstract

Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.

Figures (7)

• Figure 1: Bird's eye view of (a) convolutional and (b) attention-based networks.
• Figure 2: (a) Class-wise and (b) inter-class JS distance vs. Cohen's $\kappa$ on the Stylized ImageNet (SIN) dataset. Class-wise JS distance vs. (c) ImageNet top-1 errors and (d) SIN top-1 error.
• Figure 3: Error consistency results on SIN dataset.
• Figure 4: Shape bias for different networks for the SIN dataset geirhos2018imagenet. Vertical lines indicate averages.
• Figure 5: Error consistency results for SIN dataset before and after fine-tuning.