Generating metamers of human scene understanding

TL;DR

MetamerGen tackles the challenge of recovering human-like scene understanding by generating metamers that blend peripheral gist with fixation-driven foveal detail using a latent diffusion framework. It conditions a Stable Diffusion model with two DINOv2-based streams—$1024$ peripheral tokens and a $32$-token foveal representation after Perceiver resampling—via dedicated adapters, enabling real-time, gaze-contingent image synthesis. Behavioral experiments show that metamers conditioned on human fixations are more aligned with participants’ internal scene representations, with high-level semantic similarity (DreamSim/CLIP) most predictive of metamerism. The work demonstrates that metamers can reveal which features across low-, mid-, and high-level visual processing drive scene understanding, offering a powerful tool for cognitive science and a flexible platform for probing fixation-based theories of perception.

Abstract

Human vision combines low-resolution "gist" information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a "same" or "different" response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers' own fixated regions.

Figures (16)

• Figure 1: MetamerGen model architecture. High-resolution and blurred low-resolution images are processed through DINOv2-Base to extract patch tokens each. Foveal features are obtained by applying binary masks to high-resolution patch tokens, retaining only fixated regions. Both foveal and peripheral patch tokens are processed through separate Perceiver-based query networks that compress features into conditioning tokens compatible with Stable Diffusion's cross-attention mechanism. The resulting dual conditioning streams are integrated into the pretrained UNet for guided image denoising and generation.
• Figure 2: FID values for different input parameters of MetamerGen. Lower FID values indicate closer alignment with real images and better quality.
• Figure 3: Metameric vs. non-metameric judgments. (Left) Original images with human fixations overlaid in red and corresponding generated images judged as "same" by participants. (Right) Original images with fixations and generated images judged as "different" by participants. More examples based off of both human-fixation and random-fixation guided generations can be seen in Appendix \ref{['fig:additional_human']}
• Figure 4: Multi-level feature analysis pipeline using neurally-grounded model: (Top) Early, mid, and late network layers serve as proxies for different stages of visual processing from V1 to IT. (Bottom) Results show that as feature similarity increased at different processing stages, the proportion of participants judging generated images as metameric also increased. These effects were clearer when metamers were generated based on fixated locations (salmon) than on randomly-sampled locations (turquoise).
• Figure 5: (Left) Mid-level visual features driving metameric judgments: For metamers generated based on human-fixated locations (salmon), the preservation of monocular depth estimates in scene structure was an indicator of how more depth discrepancies yielded a decrease in metameric judgments. Additionally, when it came towards the mid-level organizational structure as seen from proto-object candidates, greater mIoU scores correlated with greater proportions of "same" metameric judgments. (Right) High-level visual features driving metameric judgments: Semantic similarity strongly predicts metameric perception, with larger DreamSim distances corresponding to reduced perceptual alignment. This result is shared with the CLIP similarity trends as well. However, these trends are less apparent when metamers were generated based on randomly-sampled locations (turquoise).