Query Smarter, Trust Better? Exploring Search Behaviours for Verifying News Accuracy

TL;DR

The paper investigates how different query-generation strategies influence the verification of online news accuracy through a three-tiered approach: reanalyzing Aslett et al.'s dataset, conducting a simulation with Llama3-based query generation across multiple strategies, and running a user study to test boost interventions. It finds that the way users formulate queries—especially using vocabulary from the source article—can reinforce belief in misinformation, while strategies that use the article's headline plus neutral context or that reformulate based on SERP results can improve the quality of retrieved evidence as measured by NewsGuard scores. Boost interventions have limited impact, though encouraging users to read the full results list and to create queries in their own words shows promise for improving assessment accuracy. Overall, the work highlights query design and interface nudges as a potential component of broader misinformation-intervention efforts, complementing media literacy and debunking approaches to enhance information evaluation.

Abstract

While it is often assumed that searching for information to evaluate misinformation will help identify false claims, recent work suggests that search behaviours can instead reinforce belief in misleading news, particularly when users generate queries using vocabulary from the source articles. Our research explores how different query generation strategies affect news verification and whether the way people search influences the accuracy of their information evaluation. A mixed-methods approach was used, consisting of three parts: (1) an analysis of existing data to understand how search behaviour influences trust in fake news, (2) a simulation of query generation strategies using a Large Language Model (LLM) to assess the impact of different query formulations on search result quality, and (3) a user study to examine how 'Boost' interventions in interface design can guide users to adopt more effective query strategies. The results show that search behaviour significantly affects trust in news, with successful searches involving multiple queries and yielding higher-quality results. Queries inspired by different parts of a news article produced search results of varying quality, and weak initial queries improved when reformulated using full SERP information. Although 'Boost' interventions had limited impact, the study suggests that interface design encouraging users to thoroughly review search results can enhance query formulation. This study highlights the importance of query strategies in evaluating news and proposes that interface design can play a key role in promoting more effective search practices, serving as one component of a broader set of interventions to combat misinformation.

Figures (5)

• Figure 1: NewsGuard Scores for Search Results against Query Position in the Session
• Figure 2: Word Overlap between Query and Article's Headline measured with the Jaccard Coefficient. The left plot represents the cases where users identified the article as false news, while the right plot represents the cases where users believed the article was truthful.
• Figure 3: Percentages of Misleading and Truthful Decisions for Queries with Varying Levels of Overlap with the Article's Headline
• Figure 4: NewsGuard Scores at different rank positions
• Figure 5: The search interface used in our study, boost shown in red in the upper left-hand corner