Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability

TL;DR

The paper investigates how the decoy effect, a cognitive bias, shapes user interactions with search engine results pages and proposes a metric to assess IR systems’ vulnerability to this bias. By analyzing logs from three public user behavior datasets, it shows decoys increase click likelihood and perceived usefulness, with the strength of the effect modulated by task difficulty and user prior knowledge. It introduces DEJA-VU, a heuristic score that balances returning highly relevant results with minimizing decoy pairs, and demonstrates its behavior across dense and sparse retrieval models, including comparisons with conventional offline metrics. The findings highlight the importance of bias-aware evaluation in IR and offer practical guidance for developing more robust, user-centered ranking and evaluation approaches.

Abstract

This study examines the decoy effect's underexplored influence on user search interactions and methods for measuring information retrieval (IR) systems' vulnerability to this effect. It explores how decoy results alter users' interactions on search engine result pages, focusing on metrics like click-through likelihood, browsing time, and perceived document usefulness. By analyzing user interaction logs from multiple datasets, the study demonstrates that decoy results significantly affect users' behavior and perceptions. Furthermore, it investigates how different levels of task difficulty and user knowledge modify the decoy effect's impact, finding that easier tasks and lower knowledge levels lead to higher engagement with target documents. In terms of IR system evaluation, the study introduces the DEJA-VU metric to assess systems' susceptibility to the decoy effect, testing it on specific retrieval tasks. The results show differences in systems' effectiveness and vulnerability, contributing to our understanding of cognitive biases in search behavior and suggesting pathways for creating more balanced and bias-aware IR evaluations.

Figures (16)

• Figure 1: An example of the decoy effect. A customer might waver between the 500ml water and the coke. Yet, with a bottle of 250ml water presenting beside the 500ml water, the customer might lean towards the 500ml water. The image is generated by DALL·E-3 and manually edited by the authors.
• Figure 2: This illustration depicts two SERPs. In the SERP on the left, the document outlined by the orange dashed line functions as a decoy for the document outlined by the light blue dashed line, forming a "decoy pair." In Experiment 1 and Experiment 2, documents such as the one outlined by the light blue dashed line, which have at least one decoy present, are designated as "target documents." Conversely, in the SERP on the right, the document outlined by the rose-colored dashed line is highly similar to the document outlined by the light blue dashed line but lacks a decoy. Such documents are classified as "control documents" in both experiments, serving as a control group for the target documents in the absence of a decoy.
• Figure 3: The relationships among the research questions in this study and the corresponding sections they relate to.
• Figure 4: An example of the SERP interfaces used for collecting user behavior in the user study datasets. Sourced from the THUIR2018 dataset Liu2018www. The original webpage was in Chinese, and the image shows the interface after being machine-translated.
• Figure 5: Data processing flow on THUIR2016, THU-KDD and THUIR2018. (1) First, we filter out decoy pairs from the search results and identify the SERP numbers where these decoy pairs are located. (2) Next, we determine documents that are similar to the target documents in the decoy pairs but do not have a decoy, serving as the control group. (3) Finally, we extract user interactions with both the target documents and control documents for further analysis.