Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation

Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, Juanzi Li

TL;DR

This work introduces MAVEN-Arg, the first all-in-one event understanding dataset that augments the MAVEN corpus with exhaustive event-argument annotations to support event detection, event argument extraction, and event relation extraction. It delivers a comprehensive schema with 162 event types and 612 argument roles, scales to 4,480 documents containing 98,591 events and 290,613 arguments, and provides document-level annotations for both entity and non-entity arguments. Empirical results show current fine-tuned EAE models and large language models struggle on MAVEN-Arg, highlighting the dataset’s challenging nature and need for methodological advances; a preliminary future-event-prediction demonstration with LLMs suggests potential with retrieval-augmented or evidence-grounded approaches. By enabling end-to-end evaluation of ED, EAE, and ERE, MAVEN-Arg offers a robust benchmark for developing practical, real-world event understanding systems and downstream applications.

Abstract

Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVEN-Arg is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVEN-Arg and codes can be obtained from https://github.com/THU-KEG/MAVEN-Argument.

MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation

TL;DR

This work introduces MAVEN-Arg, the first all-in-one event understanding dataset that augments the MAVEN corpus with exhaustive event-argument annotations to support event detection, event argument extraction, and event relation extraction. It delivers a comprehensive schema with 162 event types and 612 argument roles, scales to 4,480 documents containing 98,591 events and 290,613 arguments, and provides document-level annotations for both entity and non-entity arguments. Empirical results show current fine-tuned EAE models and large language models struggle on MAVEN-Arg, highlighting the dataset’s challenging nature and need for methodological advances; a preliminary future-event-prediction demonstration with LLMs suggests potential with retrieval-augmented or evidence-grounded approaches. By enabling end-to-end evaluation of ED, EAE, and ERE, MAVEN-Arg offers a robust benchmark for developing practical, real-world event understanding systems and downstream applications.

Abstract

Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVEN-Arg is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVEN-Arg and codes can be obtained from https://github.com/THU-KEG/MAVEN-Argument.
Paper Structure (39 sections, 6 figures, 14 tables)

This paper contains 39 sections, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Dataset Construction
  3. Event Schema Creation
  4. Schema Statistics
  5. Entity Annotation
  6. Event Argument Annotation
  7. Data Analysis
  8. Data Statistics
  9. Data Distribution
  10. Trigger-argument Distance
  11. Experiment
  12. Experimental Setup
  13. Models
  14. Evaluation Metric
  15. Experiment Results of Fine-tuned Models
  16. ...and 24 more sections

Figures (6)

  • Figure 1: Illustration for the overall event understanding, consisting of event detection, event argument extraction, and event relation extraction tasks.
  • Figure 2: Maven-Arg entity and event argument distributions. For clarity, only the top event argument roles are shown and the others are summed up in "Others".
  • Figure 3: Distribution of distances between triggers and arguments in Maven-Arg.
  • Figure 4: Mention-level F1 (%) of models on data with varying trigger-argument distances, i.e., the number of words between an event argument and its trigger.
  • Figure 5: Screenshot for the annotation platform. The trigger "headed" is selected for annotation (in the right panel) and entities are highlighted in green as the options for annotating event arguments.
  • ...and 1 more figures