Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation using GPT

Subhankar Maity, Aniket Deroy, Sudeshna Sarkar

TL;DR

This work addresses automatic MCQ generation across multiple languages by employing a language-agnostic, multi-stage prompting strategy. The MSP pipeline decomposes paraphrase generation, keyword extraction, question generation, and distractor generation, guided by chain-of-thought prompting to enable iterative reasoning without model fine-tuning. Empirical results on English, German, Hindi, and Bengali datasets show MSP outperforms single-stage prompting, with one-shot MSP notably improving distractor quality in low-resource languages, according to both automated metrics and human evaluations. The study demonstrates the practical viability of zero-shot and one-shot MSP with GPT-3.5-like models and GPT-4 for multilingual educational content generation, with future work focusing on further boosting low-resource language performance and refining evaluation metrics.

Abstract

We introduce a multi-stage prompting approach (MSP) for the generation of multiple choice questions (MCQs), harnessing the capabilities of GPT models such as text-davinci-003 and GPT-4, renowned for their excellence across various NLP tasks. Our approach incorporates the innovative concept of chain-of-thought prompting, a progressive technique in which the GPT model is provided with a series of interconnected cues to guide the MCQ generation process. Automated evaluations consistently demonstrate the superiority of our proposed MSP method over the traditional single-stage prompting (SSP) baseline, resulting in the production of high-quality distractors. Furthermore, the one-shot MSP technique enhances automatic evaluation results, contributing to improved distractor generation in multiple languages, including English, German, Bengali, and Hindi. In human evaluations, questions generated using our approach exhibit superior levels of grammaticality, answerability, and difficulty, highlighting its efficacy in various languages.

A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation using GPT

TL;DR

This work addresses automatic MCQ generation across multiple languages by employing a language-agnostic, multi-stage prompting strategy. The MSP pipeline decomposes paraphrase generation, keyword extraction, question generation, and distractor generation, guided by chain-of-thought prompting to enable iterative reasoning without model fine-tuning. Empirical results on English, German, Hindi, and Bengali datasets show MSP outperforms single-stage prompting, with one-shot MSP notably improving distractor quality in low-resource languages, according to both automated metrics and human evaluations. The study demonstrates the practical viability of zero-shot and one-shot MSP with GPT-3.5-like models and GPT-4 for multilingual educational content generation, with future work focusing on further boosting low-resource language performance and refining evaluation metrics.

Abstract

We introduce a multi-stage prompting approach (MSP) for the generation of multiple choice questions (MCQs), harnessing the capabilities of GPT models such as text-davinci-003 and GPT-4, renowned for their excellence across various NLP tasks. Our approach incorporates the innovative concept of chain-of-thought prompting, a progressive technique in which the GPT model is provided with a series of interconnected cues to guide the MCQ generation process. Automated evaluations consistently demonstrate the superiority of our proposed MSP method over the traditional single-stage prompting (SSP) baseline, resulting in the production of high-quality distractors. Furthermore, the one-shot MSP technique enhances automatic evaluation results, contributing to improved distractor generation in multiple languages, including English, German, Bengali, and Hindi. In human evaluations, questions generated using our approach exhibit superior levels of grammaticality, answerability, and difficulty, highlighting its efficacy in various languages.
Paper Structure (4 sections, 2 figures, 2 tables)

This paper contains 4 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction and Background
  2. Method
  3. Experiments
  4. Conclusion and Future Work

Figures (2)

  • Figure 1: Overview of proposed MSP approach for MCQ generation in various languages. $x \in \{\textbf{English}, German, Hindi, Bengali\}$. In this example, $x = \textbf{English}.$
  • Figure 2: An example of a generated grammatically incorrect, low-answerability question in Bengali, along with the generated highlighted correct answer option and associated distractors.