Seeing Faces in Things: A Model and Dataset for Pareidolia

TL;DR

This work investigates face pareidolia by introducing the Faces in Things dataset of 5,000 pareidolic images with human annotations and bounding boxes, enabling systematic evaluation of modern face detectors. It demonstrates a substantial performance gap between humans and state-of-the-art detectors in recognizing pareidolic faces and shows that targeted training on pareidolic or animal-face data can substantially narrow this gap. The authors propose two simple mathematical models—a Gaussian mode-based model and a higher-level feature-based model—that both predict a Goldilocks peak in pareidolia as image complexity varies, and they validate this peak with human psychophysics and machine experiments. Collectively, the dataset, transfer-learning insights, and theoretical models provide a foundation for studying and leveraging pareidolia to understand human vision and improve machine perception.

Abstract

The human visual system is well-tuned to detect faces of all shapes and sizes. While this brings obvious survival advantages, such as a better chance of spotting unknown predators in the bush, it also leads to spurious face detections. ``Face pareidolia'' describes the perception of face-like structure among otherwise random stimuli: seeing faces in coffee stains or clouds in the sky. In this paper, we study face pareidolia from a computer vision perspective. We present an image dataset of ``Faces in Things'', consisting of five thousand web images with human-annotated pareidolic faces. Using this dataset, we examine the extent to which a state-of-the-art human face detector exhibits pareidolia, and find a significant behavioral gap between humans and machines. We find that the evolutionary need for humans to detect animal faces, as well as human faces, may explain some of this gap. Finally, we propose a simple statistical model of pareidolia in images. Through studies on human subjects and our pareidolic face detectors we confirm a key prediction of our model regarding what image conditions are most likely to induce pareidolia. Dataset and Website: https://aka.ms/faces-in-things

Figures (19)

• Figure 1: You print out an exciting new computer vision paper to review, but as you sit down at your desk to start reading you knock over your coffee cup. At first, you are annoyed, but then, you laugh! The sight of the stain induces "pareidolia" in your brain: rather than an unsightly blemish, you see a happy face. In this paper we explore the phenomenon of face pareidolia: Why don't we see faces all the time? Why do we see them at all when they are clearly so different from human faces? Can a better understanding of face pareidolia help computer vision--based face detection?
• Figure 2: Examples of face pareidolia from our "Faces in Things" dataset. Faces in Things consists of five thousand images annotated with bounding boxes (shown here), and facial attributes such as perceived emotion, gender, and intentionality.
• Figure 3: Attributes of the Faces in Things Dataset. We find that 31% of faces are considered challenging to spot; faces are largely (31%) judged as happy; approximately half (47%) are judged as accidental rather than by design; animals and humans are seen in roughly equal numbers; and we observe a slight bias (16% vs 3%) towards male over female faces, similar to biases observed in prior studies wardle22wardle23cognition.
• Figure 4: The Appearance of an Average Pareidolic Face. Per-channel histogram-equalized average images for registered pareidolic faces (our Faces in Things dataset), human faces (samples from the WIDER FACE dataset yang_wider_2016), and animal faces (AnimalWeb khan2020animalweb). The average pareidolic face, while less distinct than human or animal, has surprisingly clear eye, nose, and mouth features, and vertical symmetry.
• Figure 5: Qualitative Analysis of Transfer Experiments. On a sample of held-out test images, we visualize the confident $(p>10\%)$ detections of our ground truth (red), our model fine-tuned on human faces (blue), and our model fine-tuned on animal faces (green). It is evident from these and Table \ref{['tab:results_a2p']} that fine-tuning on animal faces significantly boosts the model's ability to detect pareidolic faces.