Predicting Reaction Time to Comprehend Scenes with Foveated Scene Understanding Maps

TL;DR

This work tackles predicting human scene comprehension times by introducing F-SUM, an image-computable metric that fuses foveated vision with vision-language descriptions. By generating a gold-standard scene description with a VLM and then simulating gaze-driven foveation across many fixation points, F-SUM builds a spatial map of understanding and an aggregate score. Across 277 scenes, F-SUM correlates with human RT ($r=0.47$) and number of saccades ($r=0.51$), and with time-limited description accuracy ($r=-0.56$), outperforming baselines based on clutter, visual complexity, and language entropy. The approach is parameter-free, did not require eye-tracking for training, and highlights the central role of how spatially distributed information interacts with peripheral vision in scene understanding.

Abstract

Although models exist that predict human response times (RTs) in tasks such as target search and visual discrimination, the development of image-computable predictors for scene understanding time remains an open challenge. Recent advances in vision-language models (VLMs), which can generate scene descriptions for arbitrary images, combined with the availability of quantitative metrics for comparing linguistic descriptions, offer a new opportunity to model human scene understanding. We hypothesize that the primary bottleneck in human scene understanding and the driving source of variability in response times across scenes is the interaction between the foveated nature of the human visual system and the spatial distribution of task-relevant visual information within an image. Based on this assumption, we propose a novel image-computable model that integrates foveated vision with VLMs to produce a spatially resolved map of scene understanding as a function of fixation location (Foveated Scene Understanding Map, or F-SUM), along with an aggregate F-SUM score. This metric correlates with average (N=17) human RTs (r=0.47) and number of saccades (r=0.51) required to comprehend a scene (across 277 scenes). The F-SUM score also correlates with average (N=16) human description accuracy (r=-0.56) in time-limited presentations. These correlations significantly exceed those of standard image-based metrics such as clutter, visual complexity, and scene ambiguity based on language entropy. Together, our work introduces a new image-computable metric for predicting human response times in scene understanding and demonstrates the importance of foveated visual processing in shaping comprehension difficulty.

Figures (4)

• Figure 1: Comparison between low-effort and high-effort scene understanding. For the left scene, all the critical elements to understand the scene (baseball players) are easily identifiable in the visual periphery As a result, participants can quickly comprehend the scene regardless of fixation location, which is reflected in F-SUM; looking anywhere can obtain descriptions similar to the gold standards. In contrast, the right scene represents a high-effort scene, key elements (e.g. knife, facial expression, bag) are spatially distributed, not easily identified in the visual periphery requiring multiple fixations. Consequently, the VLM single fixation descriptions diverge from the gold standards.
• Figure 2: Overview of the F-SUM method. (1) Apply a VLM to obtain the description for the original images, deemed the gold standard description of the scene. (2) Use a foveation model to create an array of foveated images across many possible points of fixation, and apply the same VLM to return the description for each foveated rendering. (3) Construct the F-SUM based on the similarity between the gold standard and descriptions for the foveated renderings. (4) Apply the metrics using the constructed F-SUM to get a final score that measures the difficulty of understanding the scene.
• Figure 3: Overview of human psychophysics. For response time study (1), scenes were presented, participants explored the scene, and pressed the spacebar as soon as they determined that they comprehend the scene. In the the constrained saccade allotment study (2), scenes were presented for a brief time and disappeared after the participants executed 2 or 4 saccades. Participants' initial fixation for both studies was a cross toward the bottom of the image.
• Figure 4: The difference between what F-SUM could capture and what Image Complexity and Language Entropy could capture. Both Image Complexity and Language Entropy fail to predict the response time of humans on some of the images. Gold standard of the scene and human eye movement scanpath are also shown here.