Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking

Devrim Cavusoglu, Secil Sen, Ulas Sert

TL;DR

This work presents a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs), which outperforms previous methods without the need for training or fine-tuning.

Abstract

Recent advancements in Natural Language Processing (NLP) have impacted numerous sub-fields such as natural language generation, natural language inference, question answering, and more. However, in the field of question generation, the creation of distractors for multiple-choice questions (MCQ) remains a challenging task. In this work, we present a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs). Unlike previous methods, our framework relies solely on pre-trained language models and does not require additional training on specific datasets. Building upon previous research, we introduce a two-stage framework consisting of candidate generation and candidate selection. Our proposed distractor generation framework outperforms previous methods without the need for training or fine-tuning. Human evaluations confirm that our approach produces more effective and engaging distractors. The related codebase is publicly available at https://github.com/obss/disgem.

DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking

TL;DR

This work presents a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs), which outperforms previous methods without the need for training or fine-tuning.

Abstract

Recent advancements in Natural Language Processing (NLP) have impacted numerous sub-fields such as natural language generation, natural language inference, question answering, and more. However, in the field of question generation, the creation of distractors for multiple-choice questions (MCQ) remains a challenging task. In this work, we present a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs). Unlike previous methods, our framework relies solely on pre-trained language models and does not require additional training on specific datasets. Building upon previous research, we introduce a two-stage framework consisting of candidate generation and candidate selection. Our proposed distractor generation framework outperforms previous methods without the need for training or fine-tuning. Human evaluations confirm that our approach produces more effective and engaging distractors. The related codebase is publicly available at https://github.com/obss/disgem.
Paper Structure (30 sections, 3 equations, 4 figures, 11 tables)

This paper contains 30 sections, 3 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Candidate Set Generator (CSG)
  5. Generation Strategies
  6. Distractor Selector (DS)
  7. Experiments
  8. LLM Evaluations
  9. Hyperparameter Experiments
  10. Results
  11. Human Evaluation
  12. Results
  13. Discussion
  14. Conclusion
  15. Experimental Setup
  16. ...and 15 more sections

Figures (4)

  • Figure 1: The overall architecture of DisGeM. Pre-trained Language Model generates candidates, which are then filtered with two NLI models to ensure consistency among the correct answer and distractors.
  • Figure 2: Average human correctness of the cloze test grouped by the frameworks, "GOLD" represents the questions with ground truth distractors.
  • Figure 3: Average rating of the quality and difficulty of the questions by the evaluators, "GOLD" represents the questions with ground truth distractors.
  • Figure 4: Different strategies proposed for generating candidates. The prediction orders for mask tokens are (left) L2R 1-2-3-4-5, (middle) R2L 5-4-3-2-1, (right) CTL 1-5-2-4-3. "T" in blue refers to mask tokens. "C" in purple refers to the candidate tokens.