Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM with Relation Classifier for Document-Level Relation Extraction

Xingzuo Li, Kehai Chen, Yunfei Long, Min Zhang

TL;DR

This paper addresses the performance gap of large language models on document-level relation extraction by identifying NA-dominated candidate spaces as a key bottleneck. It proposes LMRC, a two-stage framework consisting of a Relation Candidate Proposal stage that uses a binary classifier with localized context pooling to filter relation-bearing entity pairs, followed by a Relation Classification stage that leverages LoRA-tuned LLMs to perform multi-class relation labeling on the reduced set. Empirical results on DocRED and Re-DocRED show LMRC substantially outperforms prior LLM-based approaches and narrows the gap with state-of-the-art BERT-based models, with ablations confirming the critical role of the pre-classifier and RC-tuning. Additional analyses on relation-frequency, out-of-domain alignment, and GPT-model integrations demonstrate the method's robustness and practicality for enhancing document-scale relation reasoning in real-world settings.

Abstract

Large language models (LLMs) have created a new paradigm for natural language processing. Despite their advancement, LLM-based methods still lag behind traditional approaches in document-level relation extraction (DocRE), a critical task for understanding complex entity relations within long context. This paper investigates the causes of this performance gap, identifying the dispersion of attention by LLMs due to entity pairs without relations as a key factor. We then introduce a novel classifier-LLM approach to DocRE. Particularly, the proposed approach begins with a classifier designed to select entity pair candidates that exhibit potential relations and then feed them to LLM for final relation classification. This method ensures that the LLM's attention is directed at relation-expressing entity pairs instead of those without relations during inference. Experiments on DocRE benchmarks reveal that our method significantly outperforms recent LLM-based DocRE models and narrows the performance gap with state-of-the-art BERT-based models.

LLM with Relation Classifier for Document-Level Relation Extraction

TL;DR

This paper addresses the performance gap of large language models on document-level relation extraction by identifying NA-dominated candidate spaces as a key bottleneck. It proposes LMRC, a two-stage framework consisting of a Relation Candidate Proposal stage that uses a binary classifier with localized context pooling to filter relation-bearing entity pairs, followed by a Relation Classification stage that leverages LoRA-tuned LLMs to perform multi-class relation labeling on the reduced set. Empirical results on DocRED and Re-DocRED show LMRC substantially outperforms prior LLM-based approaches and narrows the gap with state-of-the-art BERT-based models, with ablations confirming the critical role of the pre-classifier and RC-tuning. Additional analyses on relation-frequency, out-of-domain alignment, and GPT-model integrations demonstrate the method's robustness and practicality for enhancing document-scale relation reasoning in real-world settings.

Abstract

Large language models (LLMs) have created a new paradigm for natural language processing. Despite their advancement, LLM-based methods still lag behind traditional approaches in document-level relation extraction (DocRE), a critical task for understanding complex entity relations within long context. This paper investigates the causes of this performance gap, identifying the dispersion of attention by LLMs due to entity pairs without relations as a key factor. We then introduce a novel classifier-LLM approach to DocRE. Particularly, the proposed approach begins with a classifier designed to select entity pair candidates that exhibit potential relations and then feed them to LLM for final relation classification. This method ensures that the LLM's attention is directed at relation-expressing entity pairs instead of those without relations during inference. Experiments on DocRE benchmarks reveal that our method significantly outperforms recent LLM-based DocRE models and narrows the performance gap with state-of-the-art BERT-based models.
Paper Structure (18 sections, 6 equations, 4 figures, 8 tables)

This paper contains 18 sections, 6 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Problem Fomulation
  5. Pre-Experiments
  6. Proposed Method: LMRC
  7. Relation Candidate Proposal
  8. Relation Classification
  9. Experiments
  10. Experimental Settings
  11. Main Results
  12. Ablation Studies
  13. Fine-grained Analysis for Each Relation
  14. Out-of-Domain Relation Studies
  15. Performance of GPT models
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Statistics on the number of entity pairs in DocRED and Re-DocRED training set. NA: entity pairs that do not express any relation. Rel: entity pairs expressing relations.
  • Figure 2: Illustration of LMRC. Relation Candidate Proposal(RCP) leverages localized context pooling method_1 in the construction of a pre-processing classifier, focusing on selecting entity pairs that contain relations. Relation Classification(RC) takes the results from the previous stage to create prompt that guides fine-tuned LLaMA2 to accomplish multi-classification task.
  • Figure 3: $F_1$ scores per relation type in the DocRED development set results (darker = better). White color means that no correct predictions were made for this relation. The relations are arranged in descending order by the number of triples.
  • Figure 4: The impact of threshold $\theta$ for cosine similarity on the $F_1$ score. Left: LoRA fine-tune LLaMA2-13B-Chat. Right: LMRC with LLaMA2-13B-Chat. Both methods are conducted on the DocRED dev set.