From Recall to Reasoning: Automated Question Generation for Deeper Math Learning through Large Language Models

TL;DR

Problem: educators seek scalable, high-quality math practice content generated by GenAI. Approach: two studies evaluate current GenAI capabilities and then implement QG-DOK, a context-grounded, RAG-based framework linked to Webb's Depth of Knowledge to generate questions across cognitive levels. Key findings: plain GenAI can produce relevant questions with sufficient context, but context can reduce relevance and higher-order questions are challenging; the QG-DOK framework improves relevance, depth alignment, and lexical variety across multiple LLMs, though notation handling and mid-level depth remain areas for improvement. Significance: provides a practical, taxonomy-informed pathway to integrate GenAI into math curricula, enabling instructors to generate curriculum-aligned, progressively challenging practice problems.

Abstract

Educators have started to turn to Generative AI (GenAI) to help create new course content, but little is known about how they should do so. In this project, we investigated the first steps for optimizing content creation for advanced math. In particular, we looked at the ability of GenAI to produce high-quality practice problems that are relevant to the course content. We conducted two studies to: (1) explore the capabilities of current versions of publicly available GenAI and (2) develop an improved framework to address the limitations we found. Our results showed that GenAI can create math problems at various levels of quality with minimal support, but that providing examples and relevant content results in better quality outputs. This research can help educators decide the ideal way to adopt GenAI in their workflows, to create more effective educational experiences for students.

Figures (2)

• Figure 1: Overview of the proposed framework with two core components
• Figure 2: Ⓐ Level-1 prompt template example Ⓑ User input interface