Misinformation Exposure in the Chinese Web: A Cross-System Evaluation of Search Engines, LLMs, and AI Overviews

TL;DR

A fact-checking dataset of 12~161 Chinese Yes/No questions derived from real-world online search logs and a unified evaluation pipeline are introduced to compare three information-access paradigms: traditional search engines, standalone LLMs, and AI-generated overview modules, which reveal substantial differences in factual accuracy and topic-level variability across systems.

Abstract

Large Language Models (LLMs) are increasingly integrated into search services, providing direct answers that can reduce users' reliance on traditional result pages. Yet their factual reliability in non-English web ecosystems remains poorly understood, particularly when answering real user queries. We introduce a fact-checking dataset of 12~161 Chinese Yes/No questions derived from real-world online search logs and develop a unified evaluation pipeline to compare three information-access paradigms: traditional search engines, standalone LLMs, and AI-generated overview modules. Our analysis reveals substantial differences in factual accuracy and topic-level variability across systems. By combining this performance with real-world Baidu Index statistics, we further estimate potential exposure to incorrect factual information of Chinese users across regions. These findings highlight structural risks in AI-mediated search and underscore the need for more reliable and transparent information-access tools for the digital world.

Figures (12)

• Figure 1: Overview of our pipeline from data collection to the final fact-checking evaluation.
• Figure 2: Distribution of collected fact-checking queries across topics.
• Figure 3: Percentage of correct predictions across search engines (Baidu, Bing, Sogou), LLMs (DeepSeek, Qwen, LLaMA), and Baidu AI Overview. Error bars indicate 95% bootstrapped confidence intervals, and the red dashed line indicates the overall mean percentage of correct predictions.
• Figure 4: Percentage of correct predictions for Bing, Sogou, and Baidu on Yes/No factual queries. Error bars indicate 95% bootstrapped confidence intervals, and the red dashed line marks each engine’s overall percentage of correct predictions.
• Figure 5: Topic-level percentage of correct predictions for Bing, Sogou, and Baidu across ten domains. Each bar represents the percentage of correct predictions for a given topic, and the red dashed line marks each engine’s overall average percentage of correct predictions.