The enshittification of online search? Privacy and quality of Google, Bing and Apple in coding advice

TL;DR

This paper investigates whether online search quality remains competitive by comparing Google, Bing, and Apple on 1,467 coding-advice queries using two objective proxies: the number of trackers in first-result pages and the average rank of Stack Overflow results. It finds Bing outperforms Google and Apple in both privacy (fewer trackers) and coding-advice quality (closer to Stack Overflow). Google tends to present more ad-heavy, less coding-focused results, while Apple performs poorly on coding-related queries. The authors argue for scalable, open, time-series measurement of search quality and discuss regulatory avenues (DMA/DSA) to improve data access and accountability, highlighting implications for competition and user experience in information retrieval.

Abstract

Even though currently being challenged by ChatGPT and other large-language models (LLMs), Google Search remains one of the primary means for many individuals to find information on the internet. Interestingly, the way that we retrieve information on the web has hardly changed ever since Google was established in 1998, raising concerns as to Google's dominance in search and lack of competition. If the market for search was sufficiently competitive, then we should probably see a steady increase in search quality over time as well as alternative approaches to the Google's approach to search. However, hardly any research has so far looked at search quality, which is a key facet of a competitive market, especially not over time. In this report, we conducted a relatively large-scale quantitative comparison of search quality of 1,467 search queries relating to coding advice in October 2023. We focus on coding advice because the study of general search quality is difficult, with the aim of learning more about the assessment of search quality and motivating follow-up research into this important topic. We evaluate the search quality of Google Search, Microsoft Bing, and Apple Search, with a special emphasis on Apple Search, a widely used search engine that has never been explored in previous research. For the assessment of search quality, we use two independent metrics of search quality: 1) the number of trackers on the first search result, as a measure of privacy in web search, and 2) the average rank of the first Stack Overflow search result, under the assumption that Stack Overflow gives the best coding advice. Our results suggest that the privacy of search results is higher on Bing than on Google and Apple. Similarly, the quality of coding advice -- as measured by the average rank of Stack Overflow -- was highest on Bing.

Figures (7)

• Figure 1: iOS 16, released in September 2022, brought Apple Search to the home screen and made it more prominent than ever before. Apple Search only covers searches from within Spotlight search but not the Safari browser. At the moment, Apple has limited incentive to deploy its search engine widely in Safari because it receives major payments from Google for selecting its Search as the default in the Safari browser. The company is, however, arguably getting prepared to move away from using Google Search as a default, especially if Google should be forced to stop payments to Apple following the ongoing US court proceedings. Apple Search often gets overlooked in the debate around online search, even though it is used by hundreds of millions of Apple users every day.
• Figure 2: Compared to other search engines, Bing often embeds the code snippets from Stack Overflow directly into the search results. This suggests that developers of Bing made some intentional decisions to implement such an embedding and that they agree with our assessment that Stack Overflow regularly gives the best coding advice.
• Figure 3: An excerpt from the list of generated search queries for coding advice.
• Figure 4: Comparative analysis of top domains at rank 1 in the given search engines.
• Figure 5: Most commonly contacted third-party tracking domain by the websites behind the first search results.