The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing

TL;DR

The paper addresses how contextual semantic relevance shapes human visual processing and introduces a suite of vision-based, language-based, and combined metrics computed with deep learning models (e.g., CLIP, DETR, SentenceTransformer, Word2Vec, ConceptNet). Using a large eye-tracking dataset and generalized additive mixed models (GAMMs), it demonstrates that all metrics can predict fixation measures, with the integrated vision-language metric (total_vissem_sim) yielding the strongest predictions. The findings reveal distinct roles for vision- and language-derived cues and provide evidence for a synergistic interaction between semantic knowledge and perceptual features in guiding attention. These results advance multi-modal cognitive modeling and have implications for improving perceptual models in cognitive science and human-computer interaction.

Abstract

Semantic relevance metrics can capture both the inherent semantics of individual objects and their relationships to other elements within a visual scene. Numerous previous research has demonstrated that these metrics can influence human visual processing. However, these studies often did not fully account for contextual information or employ the recent deep learning models for more accurate computation. This study investigates human visual perception and processing by introducing the metrics of contextual semantic relevance. We evaluate semantic relationships between target objects and their surroundings from both vision-based and language-based perspectives. Testing a large eye-movement dataset from visual comprehension, we employ state-of-the-art deep learning techniques to compute these metrics and analyze their impacts on fixation measures on human visual processing through advanced statistical models. These metrics could also simulate top-down and bottom-up processing in visual perception. This study further integrates vision-based and language-based metrics into a novel combined metric, addressing a critical gap in previous research that often treated visual and semantic similarities separately. Results indicate that all metrics could precisely predict fixation measures in visual perception and processing, but with distinct roles in prediction. The combined metric outperforms other metrics, supporting theories that emphasize the interaction between semantic and visual information in shaping visual perception/processing. This finding aligns with growing recognition of the importance of multi-modal information processing in human cognition. These insights enhance our understanding of cognitive mechanisms underlying visual processing and have implications for developing more accurate computational models in fields such as cognitive science and human-computer interaction.

Figures (5)

• Figure 1: The linear combination of two waves. Note: When linearly combining two waves, the waves combine constructively, resulting in a stronger single wave. This way, the combination of a language-based metric and a vision-based metric can result in a possible stronger metric. This process is the Fourier transform.
• Figure 2: The computational method for contextual semantic relevance. Note: There are four panels. Panel A represents the image information, Panel B denotes the computation of vision-based metrics, Panel C illustrates the computation of language-based metrics, and Panel D shows the combination of one vision-based metric and one language-based metric. "Ems"=embeddings
• Figure 3: The partial effects of main metrics. Note: The first row displays the partial effects of control predictors. The second row shows the partial effects of the metrics of interest on total duration. The third row illustrates the partial effects of the metrics of interest on fixation numbers.
• Figure 4: The partial effects with random smooths. The first row displays the partial effects of the metrics on total duration, while the second row shows the partial effects on fixation number.
• Figure 5: Top-down and bottom-up processing.