Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability

TL;DR

The paper introduces BAM, a framework that synthesizes images along ANN perceptual boundaries to study and manipulate human perceptual variability. By generating a large, labeled variMNIST dataset through human trials and aligning multiple ANN models to human behavior via group- and individual-level fine-tuning, BAM enables accurate prediction of human variability. It further demonstrates adversarial generation of controversial stimuli that can guide different individuals toward divergent perceptual decisions, with validation on digits and natural images (ImageNet). The approach advances AI–human alignment and provides tools for personalized perception analysis, while suggesting extensions to broader tasks and culturally diverse populations for future work.

Abstract

Human decision-making in cognitive tasks and daily life exhibits considerable variability, shaped by factors such as task difficulty, individual preferences, and personal experiences. Understanding this variability across individuals is essential for uncovering the perceptual and decision-making mechanisms that humans rely on when faced with uncertainty and ambiguity. We present a computational framework BAM (Boundary Alignment & Manipulation framework) that combines perceptual boundary sampling in ANNs and human behavioral experiments to systematically investigate this phenomenon. Our perceptual boundary sampling algorithm generates stimuli along ANN decision boundaries that intrinsically induce significant perceptual variability. The efficacy of these stimuli is empirically validated through large-scale behavioral experiments involving 246 participants across 116,715 trials, culminating in the variMNIST dataset containing 19,943 systematically annotated images. Through personalized model alignment and adversarial generation, we establish a reliable method for simultaneously predicting and manipulating the divergent perceptual decisions of pairs of participants. This work bridges the gap between computational models and human individual difference research, providing new tools for personalized perception analysis.

Figures (33)

• Figure 1: Human perceptual variability. For the same set of stimuli, individuals often exhibit varied responses, highlighting differences in their visual perception. For instance, the digit on the left may be interpreted as a "3" by some and as a "5" by others. Similarly, the image on the right might be initially perceived as a cat by some individuals, while others may perceive it as a bird.
• Figure 2: Overview of BAM. We sample images from ANN decision boundaries using the perceptual boundary sampling algorithm for subsequent human evaluations. Our approach consists of three main components: 1. Labeling: The images generated by perceptual boundary sampling are labeled by human experiments, thus constructing the variMNIST dataset. In this process, a single image will be presented to multiple participants; 2. Aligning: Finetuning models with human behavioral data to align them with human perceptual variability at the group and individual levels, enhancing behavior prediction accuracy and aligning the models with humans; 3. Manipulating: Employing two individually aligned models, each corresponding to a specific individual, to generate images designed to elicit divergent responses between them, which are then validated through these two human participants.
• Figure 3: Sampling on perceptual boundaries. (a) The sample space can be partitioned into four distinct regions based on two classification axes. Taking the digit pair (3,7) as an example, our objective is to generate samples that induce disagreement between models A and B, as illustrated in the figure. The upper region consists of stimuli classified as "7" by model B and "3" by model A, whereas the lower region contains stimuli classified as "3" by model A and "7" by model B. The left region includes stimuli that both models classify as "3," while the right region contains stimuli that both models classify as "7." (b) Utilizing targeted controversial guidance, we constructed the variMNIST dataset. This approach employs classifier guidance on the diffusion model, directing model A toward "3" and model B toward "7," thereby constraining the generated samples to lie on perceptual decision boundaries while preserving the diffusion prior. Following model alignment with human perception, this method was further applied to generate controversial samples designed to modulate human perceptual decisions, as shown in the lower section of the right panel.
• Figure 4: Controversial guidance influence human perception. (a) Examples of three types of guidance outcome: success, bias, and failure. (b) We present the proportion and entropy distribution of our generated datasets based on handwritten digits and natural images. The upper section displays the results for handwritten digits, while the lower section corresponds to natural images. As observed, eliciting human perceptual variability is more challenging for handwritten digits, as human observers tend to exhibit high agreement on such a straightforward classification task.
• Figure 5: Human alignment results. (a) Accuracy of BaseNet, GroupNet, and IndivNet on MNIST, variMNIST, and variMNIST-i. All models performed similarly on MNIST. On variMNIST, GroupNet and IndivNet improved accuracy by $\sim$20% over BaseNet, with IndivNet outperforming GroupNet by $\sim$5% on variMNIST-i. Accuracy improved for 241 participants and decreased for 5 after inividual fine-tuning. (b) Fine-tuning results for five classifiers. On MNIST, group fine-tuning improved VIT and VGG, while others remained unchanged or declined. On variMNIST, all classifiers improved, with VIT and MLP showing the largest gains and LRM the smallest. Individual fine-tuning further improved all classifiers with the same trend. (c) For VGG, Spearman rank correlation between model and human entropy increased from $\rho=0.08$ to $\rho=0.74$ after group fine-tuning. (d) Performance of BaseNet, GroupNet, and IndivNet of varying entropy levels. The choices from selected subject for the example images are 8, 6, 9, 6, with increasing entropy levels. Here, the gray background indicates that the model's choice is inconsistent with the subject. GroupNet and IndivNet improved over BaseNet on all entropy levels, while IndivNet’s gains over GroupNet were focused on high-entropy images.