Assessing Web Search Credibility and Response Groundedness in Chat Assistants

TL;DR

This work tackles how web-search-enabled chat assistants ground their answers in external sources and the reliability of those sources. It introduces a three-stage evaluation framework—data collection with dual user roles, source credibility analysis using MBFC and fact-checking domains, and a groundedness assessment that verifies each atomic unit against explicitly cited evidence, formalized with metrics $CR$, $NCR$, $GS$, $CG$, and $NCG$. In a systematic comparison of GPT-4o, GPT-5, Perplexity, and Qwen Chat across five misinformation-prone topics and 100 claims, Perplexity achieves the highest credibility and most stable grounding, while GPT-4o exhibits higher non-credibility on sensitive topics and GPT-5 benefits from a thinking mode that boosts credibility. The study provides a reproducible benchmark and actionable guidance for improving grounding and reducing disinformation risk in web-enabled chat systems, highlighting the trade-offs between broad retrieval and source quality safeguards. It emphasizes the need for credibility-aware grounding, selective retrieval, and stronger integration with fact-checking resources to mitigate high-stakes misinformation in real-world usage.

Abstract

Chat assistants increasingly integrate web search functionality, enabling them to retrieve and cite external sources. While this promises more reliable answers, it also raises the risk of amplifying misinformation from low-credibility sources. In this paper, we introduce a novel methodology for evaluating assistants' web search behavior, focusing on source credibility and the groundedness of responses with respect to cited sources. Using 100 claims across five misinformation-prone topics, we assess GPT-4o, GPT-5, Perplexity, and Qwen Chat. Our findings reveal differences between the assistants, with Perplexity achieving the highest source credibility, whereas GPT-4o exhibits elevated citation of non-credibility sources on sensitive topics. This work provides the first systematic comparison of commonly used chat assistants for fact-checking behavior, offering a foundation for evaluating AI systems in high-stakes information environments.

Figures (9)

• Figure 1: Our methodology for evaluating chat assistants in a fact-checking context. Claims are formulated from the perspective of a Fact-Checker or Claim Believer, and chat assistants generate responses using web search. We then analyze the cited sources for credibility (Credibility Rate, Non-Credibility Rate) and measure groundedness in both credible and non-credible sources.
• Figure 2: Credibility distribution of sources cited by different chat assistants. Perplexity cites the highest proportion of high-credibility sources, while GPT-4o, GPT-5 and Qwen Chat show similar distributions of high credible sources.
• Figure 3: Distribution of groundedness across chat assistants. Blue bars show the percentage of atomic facts grounded in credible sources; the red ones show the percentage of claims supported by non-credible sources. Each distribution represents the frequency of groundedness scores across conversations.
• Figure 4: Highlighting functionality in the GPT-4o interface.
• Figure 5: GPT-5 interface with automatically activated thinking mode.