Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models

Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, Jiawei Han

TL;DR

GRE challenges traditional evaluation by producing diverse, semantically valid relations that do not match fixed references. GenRES provides a multi-dimensional framework—Topical Similarity, Uniqueness, Factualness, Granularity, and Completeness—to assess GRE outputs, employing LDA for topic modeling, embedding-derived diversity measures, LLM-based factual verification, and soft matching against gold standards. Experimental results show traditional precision/recall fail for GRE, while GenRES correlates with human judgments and reveals differences across LIBRMs and prompts, establishing a robust benchmark across document-, bag-, and sentence-level data for 14 LLMs. This framework enables more reliable, targetable evaluation of GRE systems and guides future research toward helpful, comprehensive knowledge extraction from text.

Abstract

The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE

GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models

TL;DR

GRE challenges traditional evaluation by producing diverse, semantically valid relations that do not match fixed references. GenRES provides a multi-dimensional framework—Topical Similarity, Uniqueness, Factualness, Granularity, and Completeness—to assess GRE outputs, employing LDA for topic modeling, embedding-derived diversity measures, LLM-based factual verification, and soft matching against gold standards. Experimental results show traditional precision/recall fail for GRE, while GenRES correlates with human judgments and reveals differences across LIBRMs and prompts, establishing a robust benchmark across document-, bag-, and sentence-level data for 14 LLMs. This framework enables more reliable, targetable evaluation of GRE systems and guides future research toward helpful, comprehensive knowledge extraction from text.

Abstract

The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE
Paper Structure (33 sections, 5 equations, 11 figures, 5 tables)

This paper contains 33 sections, 5 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Generative Relation Extraction
  4. GenRES
  5. Topical Similarity Score
  6. Uniqueness Score
  7. Factualness Score
  8. Granularity Score
  9. Completeness Score
  10. Experiments
  11. Datasets
  12. Implementation
  13. Performance of Different GRE Strategies
  14. Open GRE Performance of LLMs
  15. Related Works
  16. ...and 18 more sections

Figures (11)

  • Figure 1: Generative Relation Extraction (GRE): Contrasting Closed and Semi-open GRE's type constraints with Open GRE's reliance on source text alone.
  • Figure 2: GenRES framework for the evaluation of generative relation extraction (GRE).Left: An example showing the GRE process to extract triples $\mathcal{T}_{\mathcal{D}}$ from a source text $\mathcal{D}$ through prompting generative large language model. Right: illustration of sub-scores contained in GREScore regarding: Topical Similarity (§ \ref{['subsec:TS']}), Uniqueness (§ \ref{['subsec:US']}), Fatualness (§ \ref{['subsec:FS']}), Granularity (§ \ref{['subsec:GS']}), and Completeness (§ \ref{['subsec:CS']}).
  • Figure 3: Comparative Analysis of GRE Methods and Evaluation Metrics using the NYT10m Dataset. The diagram showcases the outcomes of closed, semi-open, and open Generative Relation Extraction (GRE) strategies. The distinct entity and relation spans are color-coded, with factual triples specifically highlighted. The extracted triples that affect FS, CS (soft recall), and GS are listed with the corresponding labels. We underline the ground truth labels that are inaccurate or cannot be inferred from the source text.
  • Figure 4: GRE performance of five LLMs on Wiki20m, each with five runs with random seeds.
  • Figure 5: Human Preference Evaluation (Elo Ratings) vs GenRES Evaluation on 100 Wiki20m samples.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Definition 1: Source Document
  • Definition 2: Extracted Triples