Understanding Visual Saliency of Outlier Items in Product Search

TL;DR

This study investigates how presentation features influence item outlier perception and exposure in e-commerce search results. By combining bottom-up visual saliency modeling (GBVS and Itti-Koch) with webcam-based eye-tracking in realistic online shopping tasks, it disentangles the roles of visual attributes and user goals. The findings show GBVS effectively highlights visual anomalies, but top-down factors—captured via eye-tracking—drive quick engagement with product descriptions and amplify exposure for outliers and their neighbors. These insights inform interface design and ranking fairness by illustrating how both visual cues and user intent shape attention distribution beyond position-based assumptions.

Abstract

In two-sided marketplaces, items compete for user attention, which translates to revenue for suppliers. Item exposure, indicated by the amount of attention items receive in a ranking, can be influenced by factors like position bias. Recent work suggests that inter-item dependencies, such as outlier items in a ranking, also affect item exposure. Outlier items are items that observably deviate from the other items in a ranked list. Understanding outlier items is crucial for determining an item's exposure distribution. In our previous work, we investigated the impact of different presentational features on users' perception of outlier in search results. In this work, we focus on two key questions left unanswered by our previous work: (i) What is the effect of isolated bottom-up visual factors on item outlierness in product lists? (ii) How do top-down factors influence users' perception of item outlierness in a realistic online shopping scenario? We start with bottom-up factors and employ visual saliency models to evaluate their ability to detect outlier items in product lists purely based on visual attributes. Then, to examine top-down factors, we conduct eye-tracking experiments on an online shopping task. Moreover, we employ eye-tracking to not only be closer to the real-world case but also to address the accuracy problem of reaction time in the visual search task. Our experiments show the ability of visual saliency models to detect bottom-up factors, consistently highlighting areas with strong visual contrasts. The results of our eye-tracking experiment for lists without outliers show that despite being less visually attractive, product descriptions captured attention the fastest, indicating the importance of top-down factors. In our eye-tracking experiments, we observed that outlier items engaged users for longer durations compared to non-outlier items.

Figures (10)

• Figure 1: Distribution of the RT for the (a) first and (b) second outliers in both variations of Task i.
• Figure 2: (a) Recall and (b) RT for combinations of observable features. Y-axis shows the metric of the corresponding feature and x-axis shows the second feature used in the combination.
• Figure 3: Comparative visualization of actual product lists and predicted visual saliency maps for three distinct product lists: (a), (d) and (g) show the original product list for mobile phones with an outlier image at position 3, monitors with an outlier discount tag at position 8, and of fice chairs with an outlier price at position 13, respectively. (b), (e) and (h) show the corresponding visual saliency maps using the GBVS, while (c), (f), and (i) show the maps generated by the Itti & Koch model.
• Figure 4: (a) TTFF , (b) average fixation count , and (c) average time spent by product feature category.
• Figure 5: Correlation matrix of user engagement metrics based on the eye-tracking experiment.