Reliability Criteria for News Websites
Hendrik Heuer, Elena Leah Glassman
TL;DR
This paper addresses misinformation by proposing empirically grounded reliability criteria for evaluating entire news websites rather than individual articles. It uses an integrated methodology combining contextual inquiry, think-aloud protocols, and semi-structured interviews with politically diverse laypeople (end users) in Germany and professional journalists (experts) in the US to derive 11 criteria. It distinguishes between manipulable and less manipulable criteria and shows that content, political alignment, authors, professional standards, sources, and reputation are the most robust cues used by both groups, with authors and professional standards mainly favored by experts. The work offers concrete design directions for HCI tools and platform policies to support lateral reading and scalable website-level reliability judgments, aiming to strengthen democratic discourse while preserving free speech.
Abstract
Misinformation poses a threat to democracy and to people's health. Reliability criteria for news websites can help people identify misinformation. But despite their importance, there has been no empirically substantiated list of criteria for distinguishing reliable from unreliable news websites. We identify reliability criteria, describe how they are applied in practice, and compare them to prior work. Based on our analysis, we distinguish between manipulable and less manipulable criteria and compare politically diverse laypeople as end users and journalists as expert users. We discuss 11 widely recognized criteria, including the following 6 criteria that are difficult to manipulate: content, political alignment, authors, professional standards, what sources are used, and a website's reputation. Finally, we describe how technology may be able to support people in applying these criteria in practice to assess the reliability of websites.