Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMs

TL;DR

This work combines the benefits of large language models, pairwise comparison, and Elo-based scoring to identify and quantify bias in English search query suggestions and applies this approach to the U.S. political news domain to compare bias in Google and Bing.

Abstract

Search query suggestions affect users' interactions with search engines, which then influences the information they encounter. Thus, bias in search query suggestions can lead to exposure to biased search results and can impact opinion formation. This is especially critical in the political domain. Detecting and quantifying bias in web search engines is difficult due to its topic dependency, complexity, and subjectivity. The lack of context and phrasality of query suggestions emphasizes this problem. In a multi-step approach, we combine the benefits of large language models, pairwise comparison, and Elo-based scoring to identify and quantify bias in English search query suggestions. We apply our approach to the U.S. political news domain and compare bias in Google and Bing.

Figures (2)

• Figure 1: Average bias scores (calculated on biased and unbiased query suggestions) per category for both search engines.
• Figure 2: Boxplots indicating the distribution of bias scores per category (> 8 biased texts) ordered by mean ($x$ in plot).