Distilling Multi-Scale Knowledge for Event Temporal Relation Extraction

TL;DR

This work tackles the challenge of Event Temporal Relation Extraction (ETRE) across proximity bands by introducing MulCo, an end-to-end framework that distills multi-scale knowledge between a BERT-based contextual encoder and a graph-based representation via a Subgraph-Aware Transformer (SAT). By combining Vanilla KD, Structural/Hierarchical Distillation, and a multi-scale contrastive co-distillation objective, MulCo learns a unified embedding that captures both short- and long-distance temporal cues. Extensive experiments on TB-Dense, MATRES, and TDDiscourse (TDDAuto, TDDMan) demonstrate state-of-the-art results and ablation studies validate the contribution of each component, including end-to-end training and stop-gradient guidance. Overall, MulCo offers robust ETRE across datasets with mixed proximity bands, overcoming long-input-length barriers and enabling richer temporal reasoning through coordinated BERT-GNN knowledge transfer.

Abstract

Event Temporal Relation Extraction (ETRE) is paramount but challenging. Within a discourse, event pairs are situated at different distances or the so-called proximity bands. The temporal ordering communicated about event pairs where at more remote (i.e., ``long'') or less remote (i.e., ``short'') proximity bands are encoded differently. SOTA models have tended to perform well on events situated at either short or long proximity bands, but not both. Nonetheless, real-world, natural texts contain all types of temporal event-pairs. In this paper, we present MulCo: Distilling Multi-Scale Knowledge via Contrastive Learning, a knowledge co-distillation approach that shares knowledge across multiple event pair proximity bands to improve performance on all types of temporal datasets. Our experimental results show that MulCo successfully integrates linguistic cues pertaining to temporal reasoning across both short and long proximity bands and achieves new state-of-the-art results on several ETRE benchmark datasets.

Figures (2)

• Figure 1: Overview of our proposed MulCo framework. First, we create event pairs and multi-scale event node representations from the input document using a BERT-based encoder and a GNN with a subgraph-aware transformer. Next, we apply contrastive knowledge distillation to align the event pairs with the multi-scale event node representations. Finally, we concatenate both representations and feed them into an event temporal relation classifier.
• Figure 2: Correct (blue) and incorrect (red) predictions from BERT, GNN, and MulCo on TDDAuto. The x-axis denotes all test examples of TDDAuto indexed from 0 to 4258.