Are Fact-Checking Tools Helpful? An Exploration of the Usability of Google Fact Check

TL;DR

This study evaluates the usability of Google Fact Check for retrieving fact-checking results on COVID-19 misinformation by analyzing 1,000 English false claims from the FakeCovid dataset. Using API-driven retrieval, three coders assessed relevance, and verdicts from various sources were mapped into four categories to enable cross-source comparison, complemented by LIWC-based linguistic analysis and Jaccard-based variation assessment. The findings show that only 15.8% of claims yielded results (290 total), with 94.46% of those results deemed relevant and 91.54% of relevant results labeled False or Partly False, while source reliability tended to be high but coverage remained limited; different input descriptions produced distinct results. The study recommends optimizing result quantity, potentially broadening coverage through source collaboration and suggesting input wording adjustments to elicit more useful results, while acknowledging limitations such as API-page discrepancies, topic focus, and subjectivity in coding. Overall, the work provides practical insights into the current state and limitations of fact-checking-specific search tools and informs guidance for users and tool developers seeking to improve information integrity.

Abstract

Fact-checking-specific search tools such as Google Fact Check are a promising way to combat misinformation on social media, especially during events bringing significant social influence, such as the COVID-19 pandemic and the U.S. presidential elections. However, the usability of such an approach has not been thoroughly studied. We evaluated the performance of Google Fact Check by analyzing the retrieved fact-checking results regarding 1,000 COVID-19-related false claims and found it able to retrieve the fact-checking results for 15.8% of the input claims, and the rendered results are relatively reliable. We also found that the false claims receiving different fact-checking verdicts (i.e., "False," "Partly False," "True," and "Unratable") tend to reflect diverse emotional tones, and fact-checking sources tend to check the claims in different lengths and using dictionary words to various extents. Claim variations addressing the same issue yet described differently are likely to retrieve distinct fact-checking results. We suggest that the quantities of the retrieved fact-checking results could be optimized and that slightly adjusting input wording may be the best practice for users to retrieve more useful information. This study aims to contribute to the understanding of state-of-the-art fact-checking tools and information integrity.

Figures (1)

• Figure 1: An example introducing the structure of the results on Google Fact Check. Here, an input claim obtains three results, each containing a claim assessed by a source, a source name, a fact-checking verdict, a URL to the original webpage, and a publication date. Note that input claims may receive a different number of results.