EventFull: Complete and Consistent Event Relation Annotation

TL;DR

EventFull tackles the problem of incomplete and inconsistent event relation annotation by enforcing complete coverage across temporal, causal, and coreference relations within a unified workflow. It guarantees output completeness via a layered process that combines transitive closure, consistency checks, and cross-type constraints, while remaining accessible to non-expert annotators. A pilot with three annotators on six documents shows high inter-annotator agreement and substantial reductions in manual effort, particularly for temporal and causal annotations, demonstrating the method's practicality. This approach enables the rapid creation of comprehensive, multi-relation event datasets across domains by integrating targeted intervention with automated reasoning.

Abstract

Event relation detection is a fundamental NLP task, leveraged in many downstream applications, whose modeling requires datasets annotated with event relations of various types. However, systematic and complete annotation of these relations is costly and challenging, due to the quadratic number of event pairs that need to be considered. Consequently, many current event relation datasets lack systematicity and completeness. In response, we introduce \textit{EventFull}, the first tool that supports consistent, complete and efficient annotation of temporal, causal and coreference relations via a unified and synergetic process. A pilot study demonstrates that EventFull accelerates and simplifies the annotation process while yielding high inter-annotator agreement.

Figures (7)

• Figure 1: The EventFull annotation pipeline begins with a document containing marked targeted events and proceeds through three stages: [1] Temporal Relations: annotators establish temporal relations between event pairs, supported by three processes, including pair prioritization strategy, consistency checking, and transitive relation detection (§\ref{['sec:tool-flow']}); [2] Coreference: annotators identify coreferring event mentions; [3] Causal Relations: annotators determine causal relations for pairs of events.
• Figure 2: A simple example illustrating the prioritization strategy and automatic annotation of transitive relations. The prioritization strategy incrementally presents the pairs 'accident-collided', 'collided-damage', and 'damage-responded' based on the relations that the annotator selected in each turn. The transitive relations 'accident-damage', 'accident-responded', and 'collided-responded' are detected automatically.
• Figure 3: Event Selection Annotation Step: This optional step aims to refine the set of events (detailed in §\ref{['sec:tool-flow']}) by selecting the events to be considered in subsequent steps. Annotators can access guidelines by clicking the "Event Selection Instruction" button. After categorizing all events as either event or no-event, they can proceed to the next annotation task by clicking the "Next Task" button.
• Figure 4: Temporal Relation Annotation Step: Annotators determine the temporal relations for each candidate pair based on the starting point of the events by selecting the appropriate radio button option (detailed in §\ref{['sec:tool-flow']}). Events requiring annotation are highlighted in green and red within the context, with the selected relation displayed in red in the visualization graph. All other event mentions are marked in bold within the context. The graph visualization tracks annotators' progress and allows manual selection by clicking on two nodes representing events in the graph. Annotated relations are displayed in the graph with their corresponding temporal label. Each event mention is assigned an identifier in parentheses, visible in both the context and graph, to facilitate locating events.
• Figure 5: Coreference Relation Annotation Step: Annotators determine coreference relations among all candidates annotated in the temporal step as having an equal time relation. The event mention representing the event cluster is highlighted in green, while all candidate event mentions (sharing an equal time with it) are highlighted in red. Annotators are required to check the checkbox of the red event mentions that corefer with the green one and proceed to the next group using the "Next Unhandled" button. A graph visualization tracks annotators' progress, displaying only relevant relations for this step (i.e., equal and coreference). The relations under scrutiny are highlighted in red, while not-corefer relations appear faded.