Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CMNEE: A Large-Scale Document-Level Event Extraction Dataset based on Open-Source Chinese Military News

Mengna Zhu, Zijie Xu, Kaisheng Zeng, Kaiming Xiao, Mao Wang, Wenjun Ke, Hongbin Huang

TL;DR

CMNEE introduces a large-scale, document-level dataset for open-source Chinese military news event extraction, addressing data scarcity in this domain. It comprises 17,000 documents and 29,223 manually annotated events across 8 event types and 11 argument roles, with a two-stage, multi-turn annotation workflow and explicit quality controls. A comprehensive benchmark across several model families shows that military-domain event extraction remains challenging, with trigger information and co-reference arguments significantly impacting performance. The dataset, along with code and data, provides a valuable resource to push forward research on multi-event, document-level extraction in specialized domains. CMNEE thus serves as a foundational resource to advance practical military information extraction and related decision-support applications.

Abstract

Extracting structured event knowledge, including event triggers and corresponding arguments, from military texts is fundamental to many applications, such as intelligence analysis and decision assistance. However, event extraction in the military field faces the data scarcity problem, which impedes the research of event extraction models in this domain. To alleviate this problem, we propose CMNEE, a large-scale, document-level open-source Chinese Military News Event Extraction dataset. It contains 17,000 documents and 29,223 events, which are all manually annotated based on a pre-defined schema for the military domain including 8 event types and 11 argument role types. We designed a two-stage, multi-turns annotation strategy to ensure the quality of CMNEE and reproduced several state-of-the-art event extraction models with a systematic evaluation. The experimental results on CMNEE fall shorter than those on other domain datasets obviously, which demonstrates that event extraction for military domain poses unique challenges and requires further research efforts. Our code and data can be obtained from https://github.com/Mzzzhu/CMNEE.

CMNEE: A Large-Scale Document-Level Event Extraction Dataset based on Open-Source Chinese Military News

TL;DR

CMNEE introduces a large-scale, document-level dataset for open-source Chinese military news event extraction, addressing data scarcity in this domain. It comprises 17,000 documents and 29,223 manually annotated events across 8 event types and 11 argument roles, with a two-stage, multi-turn annotation workflow and explicit quality controls. A comprehensive benchmark across several model families shows that military-domain event extraction remains challenging, with trigger information and co-reference arguments significantly impacting performance. The dataset, along with code and data, provides a valuable resource to push forward research on multi-event, document-level extraction in specialized domains. CMNEE thus serves as a foundational resource to advance practical military information extraction and related decision-support applications.

Abstract

Extracting structured event knowledge, including event triggers and corresponding arguments, from military texts is fundamental to many applications, such as intelligence analysis and decision assistance. However, event extraction in the military field faces the data scarcity problem, which impedes the research of event extraction models in this domain. To alleviate this problem, we propose CMNEE, a large-scale, document-level open-source Chinese Military News Event Extraction dataset. It contains 17,000 documents and 29,223 events, which are all manually annotated based on a pre-defined schema for the military domain including 8 event types and 11 argument role types. We designed a two-stage, multi-turns annotation strategy to ensure the quality of CMNEE and reproduced several state-of-the-art event extraction models with a systematic evaluation. The experimental results on CMNEE fall shorter than those on other domain datasets obviously, which demonstrates that event extraction for military domain poses unique challenges and requires further research efforts. Our code and data can be obtained from https://github.com/Mzzzhu/CMNEE.
Paper Structure (26 sections, 7 figures, 5 tables)

This paper contains 26 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Construction of CMNEE
  3. Data Collection and Preprocessing
  4. Event Schema Construction and Pre-labeling
  5. Human Annotation and Quality Evaluation
  6. Data Analysis of CMNEE
  7. Event Types Distribution
  8. Events Analysis
  9. Event Arguments Analysis
  10. Experiments on CMNEE
  11. Experimental setting
  12. Benchmark setting
  13. Models
  14. Evaluation
  15. Overall Experimental Results
  16. ...and 11 more sections

Figures (7)

  • Figure 1: An example from CMNEE. Each document in CMNEE is annotated with event types and involved event arguments. This document contains 6 events. We used different colors to distinguish arguments of these events and colored rectangular boxes to mark different co-reference arguments.
  • Figure 2: Main process of CMNEE construction. Firstly, crawled military news texts from websites with authority and pre-processed data. Then manually annotated them in groups after completing pre-labeling. Annotation is constantly corrected by using a two-stage multi-turns annotation approach until all the indicators reached the specified thresholds.
  • Figure 3: Data Sources
  • Figure 4: Event Type Distribution
  • Figure 5: Multi-events Distribution
  • ...and 2 more figures