When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models

TL;DR

A representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images shows that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled.

Abstract

When visual evidence is ambiguous, vision models must decide whether to interpret face-like patterns as meaningful. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior. We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images. Under a unified protocol, we evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), general object detection (YOLOv8), and face detection (RetinaFace). Our analysis reveals three mechanisms of interpretation under ambiguity. VLMs exhibit semantic overactivation, systematically pulling ambiguous non-human regions toward the Human concept, with LLaVA-1.5-7B producing the strongest and most confident over-calls, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. These results show that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. Pareidolia therefore provides a compact diagnostic and a source of ambiguity-aware hard negatives for probing and improving the semantic robustness of vision-language systems. Code will be released upon publication.

Figures (11)

• Figure 1: Face pareidolia in an electrical outlet. The visual input is unchanged, yet observers may perceive a face, illustrating how interpretation emerges under ambiguity.
• Figure 2: Example images from the FacesInThings dataset hamilton2024seeing. Red bounding boxes indicate face-like regions perceived by human observers in otherwise inanimate objects.
• Figure 3: Unified pareidolia diagnostic pipeline. VLMs (CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B) and pure vision classifier (ViT) classify annotated regions, while general object detection (YOLOv8) and face-specific detection (RetinaFace) detect faces in full images. Predictions are mapped to a common five-class space for evaluation of detection, localization, uncertainty and bias across subgroups.
• Figure 4: Detection coverage vs. localization. Box-level classifiers saturate both metrics, while YOLOv8 and RetinaFace under-respond.
• Figure 5: Detection by difficulty. Box-level models remain stable (all curves overlap at 1.0), whereas YOLOv8 and RetinaFace decline on hard, ambiguous cases.