Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI

TL;DR

This paper introduces Matchmaking for AI, a co-design methodology that engages fact-checkers, designers, and NLP researchers to translate fact-checking needs into feasible NLP tools. In a two-session workshop with 22 professionals, the authors derive 11 design ideas spanning forecasting, ambiguity resolution, reader engagement, and writing/editing support, aiming to bridge the gap between AI capabilities and real-world practice. The work contributes a translational HCI case study, a practical co-design workflow, and a set of stakeholder-informed design directions that could improve adoption and impact of NLP-based fact-checking tools. It advances both human-centered fact-checking and AI co-design by offering concrete, implementable concepts and a roadmap for future development and evaluation in domain-specific settings.

Abstract

While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs. To address this, we investigate a co-design method, Matchmaking for AI, to enable fact-checkers, designers, and NLP researchers to collaboratively identify what fact-checker needs should be addressed by technology, and to brainstorm ideas for potential solutions. Co-design sessions we conducted with 22 professional fact-checkers yielded a set of 11 design ideas that offer a "north star", integrating fact-checker criteria into novel NLP design concepts. These concepts range from pre-bunking misinformation, efficient and personalized monitoring misinformation, proactively reducing fact-checker potential biases, and collaborative writing fact-check reports. Our work provides new insights into both human-centered fact-checking research and practice and AI co-design research.

Figures (4)

• Figure 1: The Domain Specialty Canvas: fact-checkers and design facilitators map out fact-checking workflow and computational tools. While fact-checkers were thinking aloud, designers facilitated distilling important concepts and recorded them (i.e., yellow sticky notes) in the canvas, followed by asking fact-checkers to affirm, revise, or add new content (i.e., red sticky notes).
• Figure 2: An AI Probe: a half-baked AI idea consists of 1) design heuristic in the form of a textual diagram; and 2) abstracted NLP techniques in the form of an interactive Wizard-of-Oz simulation. This figure illustrates the first AI Probe (Detect checkable claims) presented to fact-checkers to help them learn the text classification technique.
• Figure 3: The Co-Design AI Canvas: fact-checkers, designers, and AI researchers collaboratively brainstorm ideas by specifying different AI elements. This figure illustrates how Idea 1 (Section \ref{['idea1']}) is formulated. First, while fact-checkers were thinking aloud, designers helped them map out different seed ideas (i.e., yellow sticky notes). Continually, fact-checkers distilled the final one (i.e., red sticky notes and diagrams in the “AI ideas”). Later, AI researchers helped depict AI information (i.e., blue sticky notes in the “AI elements”). Meanwhile, fact-checkers provided concrete examples for these AI elements. Finally, all participants brainstormed “functions and form" and “human evaluation”.
• Figure 4: A fact-checking workflow synthesized from graves2017anatomymicallef2022true to guide participants to think-aloud their experiences.