Characterizing Web Search in The Age of Generative AI

TL;DR

This study benchmarks traditional Google Organic search against four generative search engines across diverse domains to characterize how generative search outputs differ. It reveals that generative systems tend to draw from a broader and different mix of sources, with varying degrees of reliance on internal model knowledge versus retrieved web content, and they condense information into cohesive narratives rather than lists. While overall topic coverage is similar to organic search, the specific concepts surfaced vary across engines, affecting diversity and perspective. The findings underscore the need for new evaluation criteria that capture source diversity, grounding, synthesis quality, and temporal reliability in GenAI-powered search, especially for time-sensitive queries where retrieval-based systems hold an edge. The work highlights practical implications for search design and evaluation in the age of Generative AI, including how users encounter information and how up-to-date content is integrated into answers.

Abstract

The advent of LLMs has given rise to a new type of web search: Generative search, where LLMs retrieve web pages related to a query and generate a single, coherent text as a response. This output modality stands in stark contrast to traditional web search, where results are returned as a ranked list of independent web pages. In this paper, we ask: Along what dimensions do generative search outputs differ from traditional web search? We compare Google, a traditional web search engine, with four generative search engines from two providers (Google and OpenAI) across queries from four domains. Our analysis reveals intriguing differences. Most generative search engines cover a wider range of sources compared to web search. Generative search engines vary in the degree to which they rely on internal knowledge contained within the model parameters v.s. external knowledge retrieved from the web. Generative search engines surface varying sets of concepts, creating new opportunities for enhancing search diversity and serendipity. Our results also highlight the need for revisiting evaluation criteria for web search in the age of Generative AI.

Figures (43)

• Figure 1: Web search output of different search engines. \ref{['fig:search_aio', 'fig:search_organic']} show the Google search output when querying for "why is python so popular in machine learning". At the top of the results page is an AI overview produced by generative AI. The AI overview also contains links grounding the retrieved information. Below the AI Overview are the traditional web search results consisting of (often) top-10 URLs along with their titles and relevant text snippets. The response of GPT-4o with web search used as a tool is also accompanied by the supporting links (\ref{['fig:search_gpt']}).
• Figure 2: Concepts covered by different strategies for the query "What is an example of inequality?". The x-axis shows the topics discovered in the search engine outputs (Section \ref{['sec:content_analysis']}), the y-axis shows the search engines. Blue boxes indicate that the topic appeared in that engine's output. Search engines differ in the concepts they bring up.
• Figure 3: Different search engines rely on external knowledge to a different degree. GPT-Tool cites much fewer web pages than other models, followed by GPT-Search. AIO tends to cite the most web pages. The effects vary based on the dataset. The number of web pages does not show significant differences across the two countries.
• Figure 4: Different search engines select information from domains of differing ranks. Except for the Products data, the GPT models select the highest ranking sources. For the Regulatory Actions dataset, the ranks of web pages retrieved by GPT-Tool and GPT-Search are significantly higher than the other sources.
• Figure 5: Overlap between the URLs retrieved by AIO and the top-100 Organic search results for the US location. In all datasets, more than $40\%$ of the URLs retrieved by AIO are not contained in the top-100 Organic search results.