DOREMI: Optimizing Long Tail Predictions in Document-Level Relation Extraction

TL;DR

DOREMI tackles the long-tail challenge in document-level relation extraction by using an iterative, disagreement-driven active learning loop over an ensemble of core DocRE models. It selectively annotates highly disagreementful, long-tail triples to construct a denoised distant dataset that can train any DocRE model, achieving substantial gains in long-tail precision and ignPrecision with only a tiny annotation budget. The method outperforms state-of-the-art denoising baselines like UGDRE on DocRED and Re-DocRED and can be combined with frequency-focused denoising to further boost overall performance. The approach demonstrates practical impact by enabling more reliable extraction of rare relations with modest human effort, advancing knowledge base construction from text.

Abstract

Document-Level Relation Extraction (DocRE) presents significant challenges due to its reliance on cross-sentence context and the long-tail distribution of relation types, where many relations have scarce training examples. In this work, we introduce DOcument-level Relation Extraction optiMizing the long taIl (DOREMI), an iterative framework that enhances underrepresented relations through minimal yet targeted manual annotations. Unlike previous approaches that rely on large-scale noisy data or heuristic denoising, DOREMI actively selects the most informative examples to improve training efficiency and robustness. DOREMI can be applied to any existing DocRE model and is effective at mitigating long-tail biases, offering a scalable solution to improve generalization on rare relations.

Figures (8)

• Figure 1: An example of multi-entity and multi-label problems from the DocRED dataset. Subject entity "The Hitch-Hiker" (in pink) and object entity "Ida Lupino" (in blue) express two relations in the document (director and screenwriter). Other entities are in grey.
• Figure 2: Label distribution in DocRED (a) and Re-DocRED (b) annotated training dataset.
• Figure 3: The general pipeline of a DocRE model. DOREMI enhances the training dataset, optimizing long-tail relations. Once the enhanced dataset is obtained, it can be used for training any DocRE model.
• Figure 4: The DOREMI architecture including: (a) Training and Finetuning of five models on long-tail relations; (b) Disagreement Computation for each entity pair; (c) Stop Condition checks if the disagreement is below a threshold or the annotation budget is exhausted; (d) Sampling Module samples $k$ high-disagreement triples to be annotated.
• Figure 5: DOREMI flowchart. Each process or decision is linked to the corresponding computational blocks defined in Figure \ref{['fig:doremi']}.