Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors

TL;DR

This work addresses scalable, high-quality MCQ generation by grounding question creation in a hierarchical physics concept map that encodes domain knowledge, cognitive objectives, and common misconceptions. The authors compare a concept-map–guided pipeline against two baselines (base LLM and RAG), using automated validation, expert evaluation, and learner testing, and demonstrate that concept-map–driven generation achieves substantially higher high-quality rates and more discriminating distractors. Student assessments reveal that concept-map questions reduce guessing and better diagnose true conceptual understanding, albeit with lower raw accuracy due to increased difficulty. The approach offers cost and deployment advantages via a structured SQL knowledge base, enabling rapid, scalable production of diagnostic MCQs and opening pathways for expansion to other subjects and intelligent tutoring systems.

Abstract

Generating high-quality MCQs, especially those targeting diverse cognitive levels and incorporating common misconceptions into distractor design, is time-consuming and expertise-intensive, making manual creation impractical at scale. Current automated approaches typically generate questions at lower cognitive levels and fail to incorporate domain-specific misconceptions. This paper presents a hierarchical concept map-based framework that provides structured knowledge to guide LLMs in generating MCQs with distractors. We chose high-school physics as our test domain and began by developing a hierarchical concept map covering major Physics topics and their interconnections with an efficient database design. Next, through an automated pipeline, topic-relevant sections of these concept maps are retrieved to serve as a structured context for the LLM to generate questions and distractors that specifically target common misconceptions. Lastly, an automated validation is completed to ensure that the generated MCQs meet the requirements provided. We evaluate our framework against two baseline approaches: a base LLM and a RAG-based generation. We conducted expert evaluations and student assessments of the generated MCQs. Expert evaluation shows that our method significantly outperforms the baseline approaches, achieving a success rate of 75.20% in meeting all quality criteria compared to approximately 37% for both baseline methods. Student assessment data reveal that our concept map-driven approach achieved a significantly lower guess success rate of 28.05% compared to 37.10% for the baselines, indicating a more effective assessment of conceptual understanding. The results demonstrate that our concept map-based approach enables robust assessment across cognitive levels and instant identification of conceptual gaps, facilitating faster feedback loops and targeted interventions at scale.

Figures (1)

• Figure 1: Schematic of the hierarchical concept map-based MCQ generation and its validation.