Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FarsiMCQGen: a Persian Multiple-choice Question Generation Framework

Mohammad Heydari Rad, Rezvan Afari, Saeedeh Momtazi

TL;DR

FarsiMCQGen addresses the challenge of generating high-quality Persian MCQs by integrating question generation with a structured wrong-choice generation pipeline that uses fill-mask, embeddings, a POS/NER filtering, and a knowledge-graph-based ranking. It leverages PQuAD-derived Persian data and Wikipedia content to train and evaluate, culminating in a new dataset of 10,289 questions. The approach is evaluated with six large language models and human assessments, showing high validity and distractiveness of generated questions, with minimal loss when using 8-bit quantization. The work provides a practical framework and dataset to advance Persian MCQ research and educational testing tools.

Abstract

Multiple-choice questions (MCQs) are commonly used in educational testing, as they offer an efficient means of evaluating learners' knowledge. However, generating high-quality MCQs, particularly in low-resource languages such as Persian, remains a significant challenge. This paper introduces FarsiMCQGen, an innovative approach for generating Persian-language MCQs. Our methodology combines candidate generation, filtering, and ranking techniques to build a model that generates answer choices resembling those in real MCQs. We leverage advanced methods, including Transformers and knowledge graphs, integrated with rule-based approaches to craft credible distractors that challenge test-takers. Our work is based on data from Wikipedia, which includes general knowledge questions. Furthermore, this study introduces a novel Persian MCQ dataset comprising 10,289 questions. This dataset is evaluated by different state-of-the-art large language models (LLMs). Our results demonstrate the effectiveness of our model and the quality of the generated dataset, which has the potential to inspire further research on MCQs.

FarsiMCQGen: a Persian Multiple-choice Question Generation Framework

TL;DR

FarsiMCQGen addresses the challenge of generating high-quality Persian MCQs by integrating question generation with a structured wrong-choice generation pipeline that uses fill-mask, embeddings, a POS/NER filtering, and a knowledge-graph-based ranking. It leverages PQuAD-derived Persian data and Wikipedia content to train and evaluate, culminating in a new dataset of 10,289 questions. The approach is evaluated with six large language models and human assessments, showing high validity and distractiveness of generated questions, with minimal loss when using 8-bit quantization. The work provides a practical framework and dataset to advance Persian MCQ research and educational testing tools.

Abstract

Multiple-choice questions (MCQs) are commonly used in educational testing, as they offer an efficient means of evaluating learners' knowledge. However, generating high-quality MCQs, particularly in low-resource languages such as Persian, remains a significant challenge. This paper introduces FarsiMCQGen, an innovative approach for generating Persian-language MCQs. Our methodology combines candidate generation, filtering, and ranking techniques to build a model that generates answer choices resembling those in real MCQs. We leverage advanced methods, including Transformers and knowledge graphs, integrated with rule-based approaches to craft credible distractors that challenge test-takers. Our work is based on data from Wikipedia, which includes general knowledge questions. Furthermore, this study introduces a novel Persian MCQ dataset comprising 10,289 questions. This dataset is evaluated by different state-of-the-art large language models (LLMs). Our results demonstrate the effectiveness of our model and the quality of the generated dataset, which has the potential to inspire further research on MCQs.
Paper Structure (15 sections, 6 equations, 5 figures, 3 tables)

This paper contains 15 sections, 6 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Generating Question from Input Text
  5. Generating Wrong Choices
  6. Generating Wrong Choice Candidates
  7. Filtering
  8. Ranking
  9. Statistics
  10. Categorizing Based on Type
  11. Categorizing Based on Content
  12. Evaluation
  13. Conclusion
  14. Limitations
  15. Acknowledgements

Figures (5)

  • Figure 1: Question and wrong choices generation process
  • Figure 2: Generating answer sentence from question-answer pair and masking
  • Figure 3: Distribution of the questions by type
  • Figure 4: Prompt example to categorize questions based on their content. The left part is original Persian data and the right part presents its corresponding English translation.
  • Figure 5: Distribution of the questions by content