Navigating the Thin Line: Examining User Behavior in Search to Detect Engagement and Backfire Effects

TL;DR

This study investigates how search result bias and AI-predicted stance labels influence the behavior of opinionated users. Using a preregistered online experiment ($N=257$) across three debated topics, the authors manipulate two viewpoint-bias metrics ($\text{nDPB}$, $\text{nDSB}$) and display conditions (with/without stance labels) and measure clicking diversity via the Shannon index and several search-behavior metrics. They find that high bias SERPs reduce diversity and tend to depress interactions, with 19% of users abandoning the search; however, when users do engage, stance labels can increase content diversity, especially under biased conditions. These results highlight potential benefits of stance-label interventions for mitigating confirmation bias among active users, while also underlining the importance of studying user abandonment and exploring richer bias metrics and task designs for robust guidance in search systems.

Abstract

Opinionated users often seek information that aligns with their preexisting beliefs while dismissing contradictory evidence due to confirmation bias. This conduct hinders their ability to consider alternative stances when searching the web. Despite this, few studies have analyzed how the diversification of search results on disputed topics influences the search behavior of highly opinionated users. To this end, we present a preregistered user study (n = 257) investigating whether different levels (low and high) of bias metrics and search results presentation (with or without AI-predicted stances labels) can affect the stance diversity consumption and search behavior of opinionated users on three debated topics (i.e., atheism, intellectual property rights, and school uniforms). Our results show that exposing participants to (counter-attitudinally) biased search results increases their consumption of attitude-opposing content, but we also found that bias was associated with a trend toward overall fewer interactions within the search page. We also found that 19% of users interacted with queries and search pages but did not select any search results. When we removed these participants in a post-hoc analysis, we found that stance labels increased the diversity of stances consumed by users, particularly when the search results were biased. Our findings highlight the need for future research to explore distinct search scenario settings to gain insight into opinionated users' behavior.

Figures (3)

• Figure 1: SERP display conditions: A) search results only, and B) search results in A) are accompanied by AI-predicted stance labels (left) and stance overview (right).
• Figure 2: Shannon index split by SERP display and bias conditions considering I) all users (257), and II) users who issued clicks in the search platform (208). The error bars are computed using the Standard Error (SE) on the Shannon index.
• Figure 3: Users' click distribution considering different attitudes: confirming (inline with their pre-stance), neutral, and disconfirming (against their pre-stance). Conditions are grouped by I) SERP display, II) SERP bias, and III) SERP display and bias.