Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models

Yu Tian, Linh Huynh, Katerina Christhilf, Shubham Chakraborty, Micah Watanabe, Tracy Arner, Danielle McNamara

TL;DR

This work tackles the challenge of automated MCQ generation that targets distinct cognitive demands. It introduces ReQUESTA, a hybrid multi‑agent framework that decomposes MCQ authoring into planning, generation, evaluation, and post‑processing, guided by dynamic prompting and iterative refinement. Empirical evaluation against a GPT‑5 zero‑shot baseline across 20 academic passages shows that ReQUESTA produces MCQs that are more challenging, more discriminative, and more closely aligned with central concepts and learner performance, with superior distractor quality, especially for inferential items. The results demonstrate that system‑level design and agent orchestration can yield reliable, controllable artifact generation beyond simply scaling language models, and the framework’s modularity supports future extensions and domain generalization.

Abstract

Recent advances in large language models (LLMs) have made automated multiple-choice question (MCQ) generation increasingly feasible; however, reliably producing items that satisfy controlled cognitive demands remains a challenge. To address this gap, we introduce ReQUESTA, a hybrid, multi-agent framework for generating cognitively diverse MCQs that systematically target text-based, inferential, and main idea comprehension. ReQUESTA decomposes MCQ authoring into specialized subtasks and coordinates LLM-powered agents with rule-based components to support planning, controlled generation, iterative evaluation, and post-processing. We evaluated the framework in a large-scale reading comprehension study using academic expository texts, comparing ReQUESTA-generated MCQs with those produced by a single-pass GPT-5 zero-shot baseline. Psychometric analyses of learner responses assessed item difficulty and discrimination, while expert raters evaluated question quality across multiple dimensions, including topic relevance and distractor quality. Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance. Expert evaluations further indicated stronger alignment with central concepts and superior distractor linguistic consistency and semantic plausibility, particularly for inferential questions. These findings demonstrate that hybrid, agentic orchestration can systematically improve the reliability and controllability of LLM-based generation, highlighting workflow design as a key lever for structured artifact generation beyond single-pass prompting.

Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models

TL;DR

This work tackles the challenge of automated MCQ generation that targets distinct cognitive demands. It introduces ReQUESTA, a hybrid multi‑agent framework that decomposes MCQ authoring into planning, generation, evaluation, and post‑processing, guided by dynamic prompting and iterative refinement. Empirical evaluation against a GPT‑5 zero‑shot baseline across 20 academic passages shows that ReQUESTA produces MCQs that are more challenging, more discriminative, and more closely aligned with central concepts and learner performance, with superior distractor quality, especially for inferential items. The results demonstrate that system‑level design and agent orchestration can yield reliable, controllable artifact generation beyond simply scaling language models, and the framework’s modularity supports future extensions and domain generalization.

Abstract

Recent advances in large language models (LLMs) have made automated multiple-choice question (MCQ) generation increasingly feasible; however, reliably producing items that satisfy controlled cognitive demands remains a challenge. To address this gap, we introduce ReQUESTA, a hybrid, multi-agent framework for generating cognitively diverse MCQs that systematically target text-based, inferential, and main idea comprehension. ReQUESTA decomposes MCQ authoring into specialized subtasks and coordinates LLM-powered agents with rule-based components to support planning, controlled generation, iterative evaluation, and post-processing. We evaluated the framework in a large-scale reading comprehension study using academic expository texts, comparing ReQUESTA-generated MCQs with those produced by a single-pass GPT-5 zero-shot baseline. Psychometric analyses of learner responses assessed item difficulty and discrimination, while expert raters evaluated question quality across multiple dimensions, including topic relevance and distractor quality. Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance. Expert evaluations further indicated stronger alignment with central concepts and superior distractor linguistic consistency and semantic plausibility, particularly for inferential questions. These findings demonstrate that hybrid, agentic orchestration can systematically improve the reliability and controllability of LLM-based generation, highlighting workflow design as a key lever for structured artifact generation beyond single-pass prompting.
Paper Structure (46 sections, 2 figures, 7 tables)

This paper contains 46 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Classical Natural Language Processing (NLP) Approaches
  4. Transformer and LLM-based Approaches
  5. Hybrid and Multi-agent Approaches
  6. The ReQUESTA Framework
  7. System Overview and Workflow
  8. Design Rationale and Agentic Principles
  9. Decomposition and Control
  10. Iterative Refinement
  11. Parallel Generation
  12. Modular Extensibility
  13. Prompt Engineering in ReQUESTA
  14. Persona-based Prompting
  15. CoT Prompting
  16. ...and 31 more sections

Figures (2)

  • Figure 1: Overview of the ReQUESTA hybrid agentic workflow. The system implements a multi-stage, modular pipeline that combines LLM-powered agents with rule-based components to support controlled, reliable MCQ generation. LLM-powered agents are depicted as robots, while rule-based agents are shown as gears.
  • Figure 2: Mean accuracy for MCQs by question type (text-based, inferential, main idea) and question source (ReQUESTA vs. GPT-5). Error bars represent ±1 standard error. ReQUESTA-generated items show consistently lower accuracy across all question types, indicating higher difficulty compared to GPT-5-generated items.