Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generation

TL;DR

Adaptive Schema-aware Event Extraction (ASEE) jointly addresses schema selection and event extraction by integrating Schema Paraphrasing and Schema-Retrieval Augmented Generation, thereby mitigating schema hallucination and context-length issues in LLM-based EE. The authors introduce MD-SEE, a comprehensive, multi-domain, multilingual benchmark comprising 300 schemas and 12,817 training samples across 12 datasets to evaluate retrieval and extraction in tandem. Empirical results show that schema paraphrasing enhances retrieval recall across models, and end-to-end ASEE configurations achieve strong E2E-F1 scores, especially with strong retrievers and fine-tuned LLMs. The work advances practical, scalable EE with real-world schema management and provides a rigorous evaluation platform for cross-domain and cross-lingual event extraction.

Abstract

Event extraction (EE) is a fundamental task in natural language processing (NLP) that involves identifying and extracting event information from unstructured text. Effective EE in real-world scenarios requires two key steps: selecting appropriate schemas from hundreds of candidates and executing the extraction process. Existing research exhibits two critical gaps: (1) the rigid schema fixation in existing pipeline systems, and (2) the absence of benchmarks for evaluating joint schema matching and extraction. Although large language models (LLMs) offer potential solutions, their schema hallucination tendencies and context window limitations pose challenges for practical deployment. In response, we propose Adaptive Schema-aware Event Extraction (ASEE), a novel paradigm combining schema paraphrasing with schema retrieval-augmented generation. ASEE adeptly retrieves paraphrased schemas and accurately generates targeted structures. To facilitate rigorous evaluation, we construct the Multi-Dimensional Schema-aware Event Extraction (MD-SEE) benchmark, which systematically consolidates 12 datasets across diverse domains, complexity levels, and language settings. Extensive evaluations on MD-SEE show that our proposed ASEE demonstrates strong adaptability across various scenarios, significantly improving the accuracy of event extraction.

Figures (3)

• Figure 1: An example of Adaptive Schema-aware Event Extraction (ASEE) in broad domains.
• Figure 2: The architecture of ASEE comprises two primary components: Schema Paraphrasing (SP) and Schema-Retrieval Augmented Generation (SRAG, including Schema Retrieval (SR) and Schema-aware Extraction (SE)).
• Figure 3: Data Statistics of MD-SEE.