Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Shiyue Zhang, Mohit Bansal

TL;DR

The paper tackles semantic drift in text-based question generation by introducing two semantics-enhanced rewards (QPP for paraphrase likelihood and QAP for answerability) and training with reinforcement learning. It also proposes a QA-based evaluation method to assess QG quality by its ability to annotate data for QA models, aligning more closely with human judgments than traditional metrics. Leveraging back-translation-style synthetic data and two integration strategies (data filtering and mixing mini-batches), the authors demonstrate state-of-the-art QG performance and meaningful gains in semi-supervised QA on SQuAD with both BiDAF and BERT-based baselines, even without new articles. The work provides a practical semi-supervised QA pipeline and introduces evaluation methods that better capture the utility of generated questions for QA training, with implications for data augmentation in QA systems.

Abstract

Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a "semantic drift" problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this paper, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. Second, since the traditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model's ability to mimic human annotators in generating QA training data. Experiments show that our method achieves the new state-of-the-art performance w.r.t. traditional metrics, and also performs best on our QA-based evaluation metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: generate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

TL;DR

The paper tackles semantic drift in text-based question generation by introducing two semantics-enhanced rewards (QPP for paraphrase likelihood and QAP for answerability) and training with reinforcement learning. It also proposes a QA-based evaluation method to assess QG quality by its ability to annotate data for QA models, aligning more closely with human judgments than traditional metrics. Leveraging back-translation-style synthetic data and two integration strategies (data filtering and mixing mini-batches), the authors demonstrate state-of-the-art QG performance and meaningful gains in semi-supervised QA on SQuAD with both BiDAF and BERT-based baselines, even without new articles. The work provides a practical semi-supervised QA pipeline and introduces evaluation methods that better capture the utility of generated questions for QA training, with implications for data augmentation in QA systems.

Abstract

Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a "semantic drift" problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this paper, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. Second, since the traditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model's ability to mimic human annotators in generating QA training data. Experiments show that our method achieves the new state-of-the-art performance w.r.t. traditional metrics, and also performs best on our QA-based evaluation metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: generate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.

Paper Structure

This paper contains 50 sections, 6 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Question Generation
  4. Question Generation for QA
  5. Question Generation
  6. Base Model
  7. Embedding
  8. Encoder
  9. Self-attention
  10. Decoder
  11. Semantics-Reinforced Model
  12. QPP Reward
  13. QAP Reward
  14. Policy Gradient
  15. QA-Based QG Evaluation
  16. ...and 35 more sections

Figures (5)

  • Figure 1: An examples of the "semantic drift" issue in Question Generation ("Gt" is short for "ground truth").
  • Figure 2: The architecture of our semantics-reinforced QG model.
  • Figure 3: Two examples of where QPP and QAP improve in question quality evaluation.
  • Figure 4: Semi-supervised QA: First, a trained QG model generates questions from new or existing paragraphs building up a synthetic QA dataset; Second, a data filter filters out low-QAP synthetic examples and augment the rest to human-labeled QA pairs; Lastly, the QA model is trained with the enlarged QA dataset.
  • Figure 5: Some synthetic QA examples generated by our QG models.