Generating AI Literacy MCQs: A Multi-Agent LLM Approach

TL;DR

AI literacy materials and assessments for K-12 are scarce and not easily scalable. The authors present a Multi-Agent MCQ Generation System built on the LangGraph framework that uses user-defined learning objectives, grade level, and Bloom’s taxonomy levels to auto-generate MCQs, with two critique agents (Language Critique Agent and IWF Critique Agent) and a Supervisor agent enabling iterative refinement. The system produced 40 MCQs for grades K7-9, and expert evaluation showed high readability, relevance, and alignment, with some disagreements on Bloom’s level and grade-level classification; the WouldYouUseIt score (~84%) suggests classroom viability with context-dependent preferences. Overall, the work demonstrates a scalable, pedagogically grounded method to generate AI-literacy assessments and informs future classroom trials and broader applicability.

Abstract

Artificial intelligence (AI) is transforming society, making it crucial to prepare the next generation through AI literacy in K-12 education. However, scalable and reliable AI literacy materials and assessment resources are lacking. To address this gap, our study presents a novel approach to generating multiple-choice questions (MCQs) for AI literacy assessments. Our method utilizes large language models (LLMs) to automatically generate scalable, high-quality assessment questions. These questions align with user-provided learning objectives, grade levels, and Bloom's Taxonomy levels. We introduce an iterative workflow incorporating LLM-powered critique agents to ensure the generated questions meet pedagogical standards. In the preliminary evaluation, experts expressed strong interest in using the LLM-generated MCQs, indicating that this system could enrich existing AI literacy materials and provide a valuable addition to the toolkit of K-12 educators.

Figures (2)

• Figure 1: Multi-Agent MCQ Generation System.
• Figure 2: Evaluation Result