DoYouTrustAI: A Tool to Teach Students About AI Misinformation and Prompt Engineering
Phillip Driscoll, Priyanka Kumar
TL;DR
The paper addresses the risk of AI misinformation in LLM outputs and the need for AI literacy in K-12 education. It presents DoYouTrustAI, a web-based tool that generates both accurate and subtly misleading historical summaries to teach students to verify information and to understand prompt engineering. The platform combines an interactive misinformation activity with a prompt-engineering playground and uses pre/post surveys to assess learning outcomes. Implemented via a Flask backend with SQLite and OpenAI APIs, it employs a two-stage prompting strategy and immediate corrections to misinfo, arguing that prompt awareness and rapid correction can enhance digital literacy and resilience to misinformation in educational settings.
Abstract
AI, especially Large Language Models (LLMs) like ChatGPT, have rapidly developed and gained widespread adoption in the past five years, shifting user preference from traditional search engines. However, the generative nature of LLMs raises concerns about presenting misinformation as fact. To address this, we developed a web-based application that helps K-12 students enhance critical thinking by identifying misleading information in LLM responses about major historical figures. In this paper, we describe the implementation and design details of the DoYouTrustAI tool, which can be used to provide an interactive lesson which teaches students about the dangers of misinformation and how believable generative AI can make it seem. The DoYouTrustAI tool utilizes prompt engineering to present the user with AI generated summaries about the life of a historical figure. These summaries can be either accurate accounts of that persons life, or an intentionally misleading alteration of their history. The user is tasked with determining the validity of the statement without external resources. Our research questions for this work were:(RQ1) How can we design a tool that teaches students about the dangers of misleading information and of how misinformation can present itself in LLM responses? (RQ2) Can we present prompt engineering as a topic that is easily understandable for students? Our findings highlight the need to correct misleading information before users retain it. Our tool lets users select familiar individuals for testing to reduce random guessing and presents misinformation alongside known facts to maintain believability. It also provides pre-configured prompt instructions to show how different prompts affect AI responses. Together, these features create a controlled environment where users learn the importance of verifying AI responses and understanding prompt engineering.