Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model

Shang-Hsuan Chiang, Ssu-Cheng Wang, Yao-Chung Fan

TL;DR

This work targets automatic cloze distractor generation, a key component in effective language assessment. It introduces CDGP, a two-stage framework that first uses pre-trained language models to generate candidate distractors (CSG) and then ranks them with a feature-driven (DS) selector, combining PLM confidence, embedding similarities, and POS compatibility. The approach substantially outperforms state-of-the-art methods on the DGen dataset (e.g., $NDCG@10$ rising from $14.94$ to $34.17$) and yields strong results on CLOTH, with ablations showing the Candidate Set Generator as the major contributor to gains and domain-specific PLMs providing additional improvements. This highlights the practical potential of PLM-based distractor generation for scalable, high-quality cloze tests across domains, while also noting evaluation limitations and avenues for further control of distractor difficulty.

Abstract

Manually designing cloze test consumes enormous time and efforts. The major challenge lies in wrong option (distractor) selection. Having carefully-design distractors improves the effectiveness of learner ability assessment. As a result, the idea of automatically generating cloze distractor is motivated. In this paper, we investigate cloze distractor generation by exploring the employment of pre-trained language models (PLMs) as an alternative for candidate distractor generation. Experiments show that the PLM-enhanced model brings a substantial performance improvement. Our best performing model advances the state-of-the-art result from 14.94 to 34.17 (NDCG@10 score). Our code and dataset is available at https://github.com/AndyChiangSH/CDGP.

CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model

TL;DR

This work targets automatic cloze distractor generation, a key component in effective language assessment. It introduces CDGP, a two-stage framework that first uses pre-trained language models to generate candidate distractors (CSG) and then ranks them with a feature-driven (DS) selector, combining PLM confidence, embedding similarities, and POS compatibility. The approach substantially outperforms state-of-the-art methods on the DGen dataset (e.g., rising from to ) and yields strong results on CLOTH, with ablations showing the Candidate Set Generator as the major contributor to gains and domain-specific PLMs providing additional improvements. This highlights the practical potential of PLM-based distractor generation for scalable, high-quality cloze tests across domains, while also noting evaluation limitations and avenues for further control of distractor difficulty.

Abstract

Manually designing cloze test consumes enormous time and efforts. The major challenge lies in wrong option (distractor) selection. Having carefully-design distractors improves the effectiveness of learner ability assessment. As a result, the idea of automatically generating cloze distractor is motivated. In this paper, we investigate cloze distractor generation by exploring the employment of pre-trained language models (PLMs) as an alternative for candidate distractor generation. Experiments show that the PLM-enhanced model brings a substantial performance improvement. Our best performing model advances the state-of-the-art result from 14.94 to 34.17 (NDCG@10 score). Our code and dataset is available at https://github.com/AndyChiangSH/CDGP.
Paper Structure (24 sections, 1 equation, 3 figures, 9 tables)

This paper contains 24 sections, 1 equation, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. CDGP Framework
  5. Candidate Set Generator (CSG)
  6. Distractor Selector (DS)
  7. Performance Evaluation
  8. Dataset
  9. Implementation Details
  10. Evaluation Metric
  11. Automatic Score
  12. Evaluation Results
  13. Results on DGen
  14. Results on CLOTH dataset
  15. Comparing Fine-Tuning Strategy
  16. ...and 9 more sections

Figures (3)

  • Figure 1: A Cloze Test Example: the challenge to cloze test preparation lies in wrong option selection. A good wrong option selection improve the effectiveness of learner ability assessment.
  • Figure 2: CDGP Framework
  • Figure 3: The testers' feedback on the difficulty of the questions generated by CDGP (1: easiest, 5: most difficult)