Effective Yet Ephemeral Propaganda Defense: There Needs to Be More than One-Shot Inoculation to Enhance Critical Thinking

TL;DR

The paper investigates whether an AI-based propaganda detection and contextualization tool, grounded in inoculation theory and Kahneman's dual-system thinking, can produce lasting improvements in critical thinking and propaganda awareness. Through a two-phase online experiment with five inoculation-dose groups, the study finds significant CT and awareness gains during tool use but no enduring effects after the tool is removed, highlighting a reliance on the tool rather than durable skill development. The work contributes to understanding the limits of one-shot interventions and emphasizes the need for repeated exposure, booster strategies, and improved pedagogical designs to foster long-term resistance to propaganda. Practically, it informs the development of long-term resilience tools against propaganda and suggests concrete directions for future research, including spaced repetition, emotional engagement, and instructional design adjustments to promote independent critical thinking.

Abstract

In today's media landscape, propaganda distribution has a significant impact on society. It sows confusion, undermines democratic processes, and leads to increasingly difficult decision-making for news readers. We investigate the lasting effect on critical thinking and propaganda awareness on them when using a propaganda detection and contextualization tool. Building on inoculation theory, which suggests that preemptively exposing individuals to weakened forms of propaganda can improve their resilience against it, we integrate Kahneman's dual-system theory to measure the tools' impact on critical thinking. Through a two-phase online experiment, we measure the effect of several inoculation doses. Our findings show that while the tool increases critical thinking during its use, this increase vanishes without access to the tool. This indicates a single use of the tool does not create a lasting impact. We discuss the implications and propose possible approaches to improve the resilience against propaganda in the long-term.

Figures (4)

• Figure 1: Example Usage
• Figure 2: CT assessment across different conditions: (a) Comparison of CT between control/low-dose groups (reading Article 2 without tool) vs. medium/high-dose groups (reading Article 1 with tool), showing significant enhancement when using the tool; (b) Post-inoculation CT when all groups read Article 2 without tool access, demonstrating no sustained improvement; (c) Within-group comparison showing significant decline in CT when tool access was removed. Error bars represent standard error. Statistical significances: * p < 0.050 ** p < 0.010
• Figure 3: Results on propaganda awareness showing (a) self-reported increase in propaganda awareness across treatment groups, demonstrating higher perceived awareness in groups that used the tool and (b) actual accuracy in identifying propagandistic statements, showing no significant difference between groups despite varying treatments. Error bars in (b) represent the standard error.
• Figure 4: Example Usage and Architecture