Iffy-Or-Not: Extending the Web to Support the Critical Evaluation of Fallacious Texts

TL;DR

The paper addresses misinformation in online deliberation by introducing Iffy-Or-Not (ION), a Chrome extension that surfaces fallacies (highlight), facilitates diverse probes (probe), and prompts critical discussion (chat) to promote independent veracity judgments. Powered by a Meta-Llama-3-8B-Instruct LLM, ION achieves an average fallacy-detection accuracy of $0.85$, and a formative study informed design goals grounded in argumentation theory. A within-subjects user study with $N=18$ shows ION increases attention, guides search behavior, and stimulates critical thinking, though users report inaccuracies, usability issues, and potential backfiring (information overload, implied truth effect, overreliance). The results motivate mitigations (toggle controls, delayed highlights, contested outputs) and future avenues (visualizations, multimodality, and education) to broaden ION’s utility while safeguarding user agency and information integrity. Overall, ION demonstrates potential to augment critical information evaluation in online reading, supporting more informed opinion formation with careful design and context-aware deployment.

Abstract

Social platforms have expanded opportunities for deliberation with the comments being used to inform one's opinion. However, using such information to form opinions is challenged by unsubstantiated or false content. To enhance the quality of opinion formation and potentially confer resistance to misinformation, we developed Iffy-Or-Not (ION), a browser extension that seeks to invoke critical thinking when reading texts. With three features guided by argumentation theory, ION highlights fallacious content, suggests diverse queries to probe them with, and offers deeper questions to consider and chat with others about. From a user study (N=18), we found that ION encourages users to be more attentive to the content, suggests queries that align with or are preferable to their own, and poses thought-provoking questions that expands their perspectives. However, some participants expressed aversion to ION due to misalignments with their information goals and thinking predispositions. Potential backfiring effects with ION are discussed.

Figures (4)

• Figure 1: The ION extension. The access segment allows the user to enter their preferred username. The "Find Iffy Content" button in the function segment runs the fallacies detection. The description segment explains what the extension does.
• Figure 2: Technology stack for each feature of ION.
• Figure 3: Normalized confusion matrix for the full data ($N = 630$)
• Figure 4: Normalized confusion matrix for the subset data ($N = 517$)