Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explicit Diversity Conditions for Effective Question Answer Generation with Large Language Models

Vikas Yadav, Hyuk Joon Kwon, Vijay Srinivasan, Hongxia Jin

TL;DR

This work presents explicit diversity conditions for QAG, focusing on spatial aspects, question types, and entities, substantially increasing diversity in QA generation by showing significant improvements in downstream QA task over existing widely adopted implicit diversity techniques.

Abstract

Question Answer Generation (QAG) is an effective data augmentation technique to improve the accuracy of question answering systems, especially in low-resource domains. While recent pretrained and large language model-based QAG methods have made substantial progress, they face the critical issue of redundant QA pair generation, affecting downstream QA systems. Implicit diversity techniques such as sampling and diverse beam search are proven effective solutions but often yield smaller diversity. We present explicit diversity conditions for QAG, focusing on spatial aspects, question types, and entities, substantially increasing diversity in QA generation. Our work emphasizes the need of explicit diversity conditions for generating diverse question-answer synthetic data by showing significant improvements in downstream QA task over existing widely adopted implicit diversity techniques. In particular, generated QA pairs from explicit diversity conditions when used to train the downstream QA model results in an average 4.1% exact match and 4.5% F1 improvement over QAG from implicit sampling techniques on SQuADDU. Our work emphasizes the need for explicit diversity conditions even more in low-resource datasets (SubjQA), where average downstream QA performance improvements are around 12% EM.

Explicit Diversity Conditions for Effective Question Answer Generation with Large Language Models

TL;DR

This work presents explicit diversity conditions for QAG, focusing on spatial aspects, question types, and entities, substantially increasing diversity in QA generation by showing significant improvements in downstream QA task over existing widely adopted implicit diversity techniques.

Abstract

Question Answer Generation (QAG) is an effective data augmentation technique to improve the accuracy of question answering systems, especially in low-resource domains. While recent pretrained and large language model-based QAG methods have made substantial progress, they face the critical issue of redundant QA pair generation, affecting downstream QA systems. Implicit diversity techniques such as sampling and diverse beam search are proven effective solutions but often yield smaller diversity. We present explicit diversity conditions for QAG, focusing on spatial aspects, question types, and entities, substantially increasing diversity in QA generation. Our work emphasizes the need of explicit diversity conditions for generating diverse question-answer synthetic data by showing significant improvements in downstream QA task over existing widely adopted implicit diversity techniques. In particular, generated QA pairs from explicit diversity conditions when used to train the downstream QA model results in an average 4.1% exact match and 4.5% F1 improvement over QAG from implicit sampling techniques on SQuADDU. Our work emphasizes the need for explicit diversity conditions even more in low-resource datasets (SubjQA), where average downstream QA performance improvements are around 12% EM.

Paper Structure

This paper contains 12 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Explicit diversity prompts
  5. Combined prompts
  6. Implicit Sampling and Decoding
  7. Hyperparameters:
  8. Evaluation
  9. Overlap and Coverage Analyses
  10. Conclusion:
  11. Future Work
  12. Ethical consideration

Figures (1)

  • Figure 1: A sample input passage and QA pairs generated by human annotators, nucleus sampling based beam search and our explicit diversity prompting techniques. Different colors in the document text depict the 5 different positions. QA pairs from specific positions are depicted in the same font color and WH question types are indicated in blue bounding boxes. Example of each explicit diversity prompts are shown in the top block.