Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer

TL;DR

The paper examines Japanese document-level relation extraction (DocRE) by leveraging English DocRE resources through cross-lingual transfer. It first constructs an automatically transferred dataset (Re-DocREDja) and demonstrates recall gaps when applied to real Japanese text due to topic shifts and surface-structure differences. To address this, the authors introduce JacRED, a semi-automatic, edit-based dataset built with human annotators guided by machine recommendations trained on Re-DocREDja, achieving roughly a 50% reduction in human edits compared with knowledge-base-only approaches. Experiments reveal that cross-lingual transfer helps annotation but faces significant limitations for direct DocRE performance and cross-lingual transfer, establishing JacRED as a challenging yet valuable benchmark for Japanese DocRE and cross-lingual research.

Abstract

Document-level Relation Extraction (DocRE) is the task of extracting all semantic relationships from a document. While studies have been conducted on English DocRE, limited attention has been given to DocRE in non-English languages. This work delves into effectively utilizing existing English resources to promote DocRE studies in non-English languages, with Japanese as the representative case. As an initial attempt, we construct a dataset by transferring an English dataset to Japanese. However, models trained on such a dataset suffer from low recalls. We investigate the error cases and attribute the failure to different surface structures and semantics of documents translated from English and those written by native speakers. We thus switch to explore if the transferred dataset can assist human annotation on Japanese documents. In our proposal, annotators edit relation predictions from a model trained on the transferred dataset. Quantitative analysis shows that relation recommendations suggested by the model help reduce approximately 50% of the human edit steps compared with the previous approach. Experiments quantify the performance of existing DocRE models on our collected dataset, portraying the challenges of Japanese and cross-lingual DocRE.

Figures (5)

• Figure 1: Overview of the proposed annotation scheme. src and tgt represent the source and target language, respectively. Previous works require 4 human edit steps to reach the final annotation, while ours only require 2.
• Figure 2: Transferring Re-DocRED from English into Japanese. We post-edit the translation to detach case markers from entity spans.
• Figure 3: Cases where the model trained on Re-DocREDja failed to predict. Documents are shown as partial for better visibility. Note that English translations are provided only for reference, while predictions are actually done on Japanese texts.
• Figure 4: Interface for relation annotation. English translations are provided on the right for reference. In this example, the annotator decides whether (Helen Craig McCullough, Employer, the University of California, Berkeley) holds or not. Entity mentions connected with Coref are coreferences of each other.
• Figure 5: An example of the prompt used for the in-context learning of GPT-3.5 and GPT-4.