AutoMCQ -- Automatically Generate Code Comprehension Questions using GenAI

TL;DR

Code comprehension gaps in novice programmers are highlighted, especially as GenAI tools become prevalent in software development education. The authors present AutoMCQ, a system that automatically generates bespoke multiple-choice questions about student submissions by coupling automated unit testing with AI-driven prompts via GPT-4o mini, integrated into CodeRunner within Moodle. The system explicitly marks AI-generated questions to enable manual review and formative use rather than summative assessment. Early results indicate positive student reception and small gains in assessment metrics, but the authors call for larger, controlled studies to validate effectiveness and refine question quality.

Abstract

Students often do not fully understand the code they have written. This sometimes does not become evident until later in their education, which can mean it is harder to fix their incorrect knowledge or misunderstandings. In addition, being able to fully understand code is increasingly important in a world where students have access to generative artificial intelligence (GenAI) tools, such as GitHub Copilot. One effective solution is to utilise code comprehension questions, where a marker asks questions about a submission to gauge understanding, this can also have the side effect of helping to detect plagiarism. However, this approach is time consuming and can be difficult and/or expensive to scale. This paper introduces AutoMCQ, which uses GenAI for the automatic generation of multiple-choice code comprehension questions. This is integrated with the CodeRunner automated assessment platform.

Figures (3)

• Figure 1: System Architecture
• Figure 2: Building Class
• Figure 3: Generated Questions