Unexpected Knowledge: Auditing Wikipedia and Grokipedia Search Recommendations

TL;DR

The paper conducts a large-scale, cross-platform audit of search recommendations on Wikipedia and Grokipedia, revealing that both systems often surface unexpectedly related or unrelated results from neutral inputs and that their recommendation sets diverge considerably. By combining single-step and multi-stage analyses, the authors quantify semantic alignment, topical distributions, and topic-transition dynamics, showing systematic differences in how content is surfaced and how exploration paths evolve. These findings highlight that encyclopedic search interfaces can guide users toward unforeseen content, with implications for information access and platform design in both human-curated and AI-generated knowledge bases.

Abstract

Encyclopedic knowledge platforms are key gateways through which users explore information online. The recent release of Grokipedia, a fully AI-generated encyclopedia, introduces a new alternative to traditional, well-established platforms like Wikipedia. In this context, search engine mechanisms play an important role in guiding users exploratory paths, yet their behavior across different encyclopedic systems remains underexplored. In this work, we address this gap by providing the first comparative analysis of search engine in Wikipedia and Grokipedia. Using nearly 10,000 neutral English words and their substrings as queries, we collect over 70,000 search engine results and examine their semantic alignment, overlap, and topical structure. We find that both platforms frequently generate results that are weakly related to the original query and, in many cases, surface unexpected content starting from innocuous queries. Despite these shared properties, the two systems often produce substantially different recommendation sets for the same query. Through topical annotation and trajectory analysis, we further identify systematic differences in how content categories are surfaced and how search engine results evolve over multiple stages of exploration. Overall, our findings show that unexpected search engine outcomes are a common feature of both the platforms, even though they exhibit discrepancies in terms of topical distribution and query suggestions.

Figures (6)

• Figure 1: Cosine similarity between queries and recommendations provided by Grokipedia and Wikipedia.
• Figure 2: Jaccard similarity between Grokipedia and Wikipedia recommendation sets.
• Figure 3: Topical distribution of recommendations provided by Grokipedia and Wikipedia.
• Figure 4: Proportion of search engine recommendations with a given topic (Recommendation), generated from a given query topic (Query). Blank cells indicate $0$-values.
• Figure 5: Topical multi-stage recommendation graphs. Nodes represent the encountered topics, while edges indicate transitions between topics. Edge weights show the probability of transition from a topic to another. To ensure readability, only the 3-most common topical transitions are visualized.