Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection

Yi-Fan Lu, Xian-Ling Mao, Tian Lan, Tong Zhang, Yu-Shi Zhu, Heyan Huang

TL;DR

SEOE tackles the evaluation bottleneck in Open Domain Event Detection by constructing a large, 7-domain benchmark with 564 event types and a cost-effective annotation strategy, paired with a semantic evaluation metric that uses LLMs to measure semantic F1 against gold labels. The benchmark is built through ontology integration, fine-grained definitions, latent type recognition, and supplementary annotations, reinforced by nucleus sampling to enhance reliability. Semantic evaluation leverages grouped fine-grained definitions and LLM-based matching to achieve high correlation with human judgments, with GPT-4o performing best among tested agents. Experimental results reveal that ODED remains challenging, particularly for predicting event types and their definitions, and offer insights into error patterns and the trade-offs between extraction accuracy and coverage. The work demonstrates practical scalability and provides a framework for more representative, semantic-driven evaluation in open-domain event extraction.

Abstract

Automatic evaluation for Open Domain Event Detection (ODED) is a highly challenging task, because ODED is characterized by a vast diversity of un-constrained output labels from various domains. Nearly all existing evaluation methods for ODED usually first construct evaluation benchmarks with limited labels and domain coverage, and then evaluate ODED methods using metrics based on token-level label matching rules. However, this kind of evaluation framework faces two issues: (1) The limited evaluation benchmarks lack representatives of the real world, making it difficult to accurately reflect the performance of various ODED methods in real-world scenarios; (2) Evaluation metrics based on token-level matching rules fail to capture semantic similarity between predictions and golden labels. To address these two problems above, we propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection (SEOE) by constructing a more representative evaluation benchmark and introducing a semantic evaluation metric. Specifically, our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains, with a cost-effective supplementary annotation strategy to ensure the benchmark's representativeness. The strategy also allows for the supplement of new event types and domains in the future. Then, the proposed SEOE leverages large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels to enhance the reliability of the evaluation. Extensive experiments validate the representatives of the benchmark and the reliability of the semantic evaluation metric. Existing ODED methods are thoroughly evaluated, and the error patterns of predictions are analyzed, revealing several insightful findings.

SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection

TL;DR

SEOE tackles the evaluation bottleneck in Open Domain Event Detection by constructing a large, 7-domain benchmark with 564 event types and a cost-effective annotation strategy, paired with a semantic evaluation metric that uses LLMs to measure semantic F1 against gold labels. The benchmark is built through ontology integration, fine-grained definitions, latent type recognition, and supplementary annotations, reinforced by nucleus sampling to enhance reliability. Semantic evaluation leverages grouped fine-grained definitions and LLM-based matching to achieve high correlation with human judgments, with GPT-4o performing best among tested agents. Experimental results reveal that ODED remains challenging, particularly for predicting event types and their definitions, and offer insights into error patterns and the trade-offs between extraction accuracy and coverage. The work demonstrates practical scalability and provides a framework for more representative, semantic-driven evaluation in open-domain event extraction.

Abstract

Automatic evaluation for Open Domain Event Detection (ODED) is a highly challenging task, because ODED is characterized by a vast diversity of un-constrained output labels from various domains. Nearly all existing evaluation methods for ODED usually first construct evaluation benchmarks with limited labels and domain coverage, and then evaluate ODED methods using metrics based on token-level label matching rules. However, this kind of evaluation framework faces two issues: (1) The limited evaluation benchmarks lack representatives of the real world, making it difficult to accurately reflect the performance of various ODED methods in real-world scenarios; (2) Evaluation metrics based on token-level matching rules fail to capture semantic similarity between predictions and golden labels. To address these two problems above, we propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection (SEOE) by constructing a more representative evaluation benchmark and introducing a semantic evaluation metric. Specifically, our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains, with a cost-effective supplementary annotation strategy to ensure the benchmark's representativeness. The strategy also allows for the supplement of new event types and domains in the future. Then, the proposed SEOE leverages large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels to enhance the reliability of the evaluation. Extensive experiments validate the representatives of the benchmark and the reliability of the semantic evaluation metric. Existing ODED methods are thoroughly evaluated, and the error patterns of predictions are analyzed, revealing several insightful findings.

Paper Structure

This paper contains 36 sections, 8 equations, 5 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Event Detection
  4. LLMs for Automatic Benchmark Construction
  5. Evaluation for Event Detection
  6. ODED Task Formulation
  7. Open Domain Event Benchmark
  8. Benchmark Construction
  9. Step 1: Ontology and Data Integration
  10. Step 2: Fine-grained Definitions Generation
  11. Step 3: Latent Event Types Recognition
  12. Step 4: Supplementary Annotations
  13. Benchmark Reliability Enhanced by Nucleus Sampling
  14. Benchmark Statistics and Scalability
  15. Semantic Evaluation by LLMs
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Comparison between previous ODED evaluation frameworks and our proposed SEOE.
  • Figure 2: The pipelined construction process of proposed open domain event detection evaluation dataset.
  • Figure 3: Distribution of SEOE benchmark.
  • Figure 4: T-SNE JMLR:v9:vandermaaten08a visualization results of our constructed groups. To maintain conciseness, we only present a subset of groups.
  • Figure 5: Error Pattern Distribution. All data are randomly sampled from the main experiment.