Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidence of political bias in search engines and language models before major elections

Íris Damião, Paulo Almeida, João Franco, Nuno Santos, Pedro C. Magalhães, Joana Gonçalves-Sá

Abstract

Search engines (SEs) and large language models (LLMs) are central to political information access, yet their algorithmic decisions and potential underlying biases remain underexplored. We developed a standardized, privacy-preserving, bot-and-proxy methodology to audit four SEs and two LLMs before the 2024 European Parliament and US presidential elections. We collected answers to approximately 4,360 queries related to elections in five EU countries and 15 US counties, identified political entities and topics in those answers, and mapped them to ideological positions (EU) or issue associations (US). In Europe, SE results disproportionately mentioned far-right entities beyond levels expected from polls, past elections, or media salience. In the US, Google strongly favored topics more important to Republican voters, while other search engines favored issues more relevant to Democrats. LLMs responses were more balanced, although there is evidence of overrepresentation of far-right (and Green) entities. These results show evidence of bias and open important discussions on how even small skews in widely used platforms may influence democratic processes, calling for systematic audits of their outputs.

Evidence of political bias in search engines and language models before major elections

Abstract

Search engines (SEs) and large language models (LLMs) are central to political information access, yet their algorithmic decisions and potential underlying biases remain underexplored. We developed a standardized, privacy-preserving, bot-and-proxy methodology to audit four SEs and two LLMs before the 2024 European Parliament and US presidential elections. We collected answers to approximately 4,360 queries related to elections in five EU countries and 15 US counties, identified political entities and topics in those answers, and mapped them to ideological positions (EU) or issue associations (US). In Europe, SE results disproportionately mentioned far-right entities beyond levels expected from polls, past elections, or media salience. In the US, Google strongly favored topics more important to Republican voters, while other search engines favored issues more relevant to Democrats. LLMs responses were more balanced, although there is evidence of overrepresentation of far-right (and Green) entities. These results show evidence of bias and open important discussions on how even small skews in widely used platforms may influence democratic processes, calling for systematic audits of their outputs.
Paper Structure (58 sections, 9 equations, 32 figures, 33 tables)

This paper contains 58 sections, 9 equations, 32 figures, 33 tables.

Table of Contents

  1. Introduction
  2. Results
  3. Search Engines and Large Language Models offer political content
  4. Search Engines and Large Language Models overrepresent certain political leanings
  5. The observed leaning corresponds to a systematic bias
  6. Discussion
  7. Methods
  8. Data Collection
  9. Webcrawlers
  10. Audits
  11. Search Queries
  12. Collected Outputs
  13. SE results category classification
  14. Entities Extraction
  15. Mention counts
  16. ...and 43 more sections

Figures (32)

  • Figure 1: Methodological scheme.(1) Bots collected data from search engines (Google, Bing, DuckDuckGo, Yahoo) and LLMs (Copilot, ChatGPT), from different locations within the European Union (Austria - AU, Germany - DE, Ireland - IR, Poland - PL, and Portugal - PT) or from different counties in the United States of America. For LLMs, location was simulated by prompting the model in the most spoken language of each country (EU only), rather than through actual geolocation. (2) Political entities were extracted from the results collected in (1), i.e. from URLs and headlines for SEs and from textual answers for LLMs. (3) These entities were mapped to their ideological leanings using their placement on political spectra in the case of the EU, or according to the issues considered more important to Democratic or Republican voters in the case of the US. (4) The observed leanings were compared with external factors (media salience, polls, prior electoral results).
  • Figure 2: Search Engine and LLM results classified by political mentions. Dark gray bars show the proportion of SE results (A, B, C) or LLM textual answers (D, E, F) containing relevant mentions (political entities -- A, B, D and E -- or issues -- C and F). The plots on the right (color bars) show, for the results that mention at least one political entity, the average proportion of mentions by political leaning or issue categories. In A and D colors represent broad EU political families (Radical Left: dark red; Mainstream Left: red; Center/Greens: green; Mainstream Right: blue; Radical Right: dark blue). In B and D colors represent mentions to either Republican (red) or Democratic (blue) entities. In C and F colors correspond to the leaning of the mentioned issues (Rep ++ = much more important for Republicans, dark red; Rep + = slightly more important for Republicans, red; Dem + = slightly more important for Democrats, blue; Dem ++ = much more important for Democrats, dark blue, Neutral = equally important to both, gray). In A and D (EU) each dot represents a country; in B and C (US) each dot represents a county. Error bars show 95% confidence intervals computed via 1,000 bootstrap resamples (sampling with replacement from all mentions). Statistical details in Tables \ref{['tab:agg_uni_eu']} - \ref{['tab:uni_global_llm']} of Appendix.
  • Figure 3: Comparison with external factors. (A) Explanatory scheme of the differences displayed in (B - G). Star = % in Algorithms (SEs or LLMs) - % Average attention in Media; Circle = % in Algorithms - % Previous Results; Cross = % Algorithms - % Polls; Triangle = % Algorithms - % Average importance of the Issue (US only), according to surveys pew2024issuesyougov2025issues. Positive values indicate overrepresentation by the algorithms, while negative values indicate underrepresentation. (B - D) Google attention to EU and US political leanings vs. Media, Previous results and polls. (E - G) ChatGPTattention to EU and US political leanings and issues compared with the same external factors. Z-scores for each comparison and political leaning are detailed in Appendix, Tables\ref{['tab:agg_ext_eu']}-\ref{['tab:ext_global_llm']}.
  • Figure 4: Leaning values for key topics, combining Pew Research pew2024issuesand YouGov yougov2025issues data collected in October 2024.
  • Figure A1: EU Parliament Election external source data per country. Column 1 reports the projected number of seats for each political leaning based on EuObserver polling during the week preceding the election. Column 2 presents the distribution of seats obtained by each political leaning in the 2019 constitutive session of the EU Parliament. Column 3 shows the actual 2024 election results for the five countries included in this study.
  • ...and 27 more figures