Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Better Question Generation in QA-based Event Extraction

Zijin Hong, Jian Liu

TL;DR

This work tackles QA-based event extraction by addressing the critical role of question quality. It defines four criteria for high-quality questions and develops RLQG, a reinforcement learning framework that combines inverse prompting and QA rewards to produce context-dependent, fluent, generalizable, and indicative questions. The method integrates a supervised QG module with beam search augmentation and uses PPO-based refinement to optimize question quality, evaluated on ACE 2005 and RAMS where it outperforms strong baselines, especially in data-scarce scenarios. An off-the-shelf QA model then answers the generated questions to complete the EE task, with results showing notable gains and robustness across architectures. The work advances QA-based EE by enabling high-quality, adaptable question generation with practical impact for information extraction in low-resource settings, and provides code for replication and further research.

Abstract

Event Extraction (EE) is an essential information extraction task that aims to extract event-related information from unstructured texts. The paradigm of this task has shifted from conventional classification-based methods to more contemporary question-answering-based (QA-based) approaches. However, in QA-based EE, the quality of the questions dramatically affects the extraction accuracy, and how to generate high-quality questions for QA-based EE remains a challenge. In this work, to tackle this challenge, we suggest four criteria to evaluate the quality of a question and propose a reinforcement learning method, RLQG, for QA-based EE that can generate generalizable, high-quality, and context-dependent questions and provides clear guidance to QA models. The extensive experiments conducted on ACE and RAMS datasets have strongly validated our approach's effectiveness, which also demonstrates its robustness in scenarios with limited training data. The corresponding code of RLQG is released for further research.

Towards Better Question Generation in QA-based Event Extraction

TL;DR

This work tackles QA-based event extraction by addressing the critical role of question quality. It defines four criteria for high-quality questions and develops RLQG, a reinforcement learning framework that combines inverse prompting and QA rewards to produce context-dependent, fluent, generalizable, and indicative questions. The method integrates a supervised QG module with beam search augmentation and uses PPO-based refinement to optimize question quality, evaluated on ACE 2005 and RAMS where it outperforms strong baselines, especially in data-scarce scenarios. An off-the-shelf QA model then answers the generated questions to complete the EE task, with results showing notable gains and robustness across architectures. The work advances QA-based EE by enabling high-quality, adaptable question generation with practical impact for information extraction in low-resource settings, and provides code for replication and further research.

Abstract

Event Extraction (EE) is an essential information extraction task that aims to extract event-related information from unstructured texts. The paradigm of this task has shifted from conventional classification-based methods to more contemporary question-answering-based (QA-based) approaches. However, in QA-based EE, the quality of the questions dramatically affects the extraction accuracy, and how to generate high-quality questions for QA-based EE remains a challenge. In this work, to tackle this challenge, we suggest four criteria to evaluate the quality of a question and propose a reinforcement learning method, RLQG, for QA-based EE that can generate generalizable, high-quality, and context-dependent questions and provides clear guidance to QA models. The extensive experiments conducted on ACE and RAMS datasets have strongly validated our approach's effectiveness, which also demonstrates its robustness in scenarios with limited training data. The corresponding code of RLQG is released for further research.
Paper Structure (43 sections, 10 equations, 4 figures, 15 tables, 1 algorithm)

This paper contains 43 sections, 10 equations, 4 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. QA-Based Event Extraction
  4. Prompt Engineering for LLMs
  5. Proposed Method
  6. Question Generation Module
  7. SFT over Template Questions.
  8. Beam Search Augmentation.
  9. RL for QG Refinement
  10. Inverse Prompting Reward.
  11. Question Answering Reward.
  12. Refining with RL.
  13. Question Answering Module
  14. Experimental Setups
  15. Datasets
  16. ...and 28 more sections

Figures (4)

  • Figure 1: An EE example from ACE dataset, with "firefight" being the event trigger and "Marines" being the event argument fulfilling the Attacker role.
  • Figure 2: The overview of our proposed method, comprises: 1) Training phase, including supervised fine-tuning and reinforcement learning refining for a question generation model. 2) Testing phase, a final (off-the-shelf) question-answering model predicts the final answer based on the given context and question.
  • Figure 3: Experimental results in the ACE dataset for the data-scarce scenario. The x-axis represents different ratios of training data, y-axis is the value of the metric.
  • Figure 4: The performance with exact match accuracy on different QA model (a) LLaMA-2-13b-Chat (5shot) (identical to previous settings) (b) LLaMA-2-13b-Chat (0shot) (c) GPT-4 (5shot) (d) GPT-4 (0shot)