Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction

TL;DR

This work tackles zero-shot document-level relation triplet extraction by distilling latent relational facts from large language models. It introduces GenRDK, a pipeline that uses chain-of-retrieval prompts to generate labeled documents with unseen relations, a pre-denoising model to produce pseudo labels, and a consistency-guided cross-document denoising process that builds and fuse knowledge graphs to prune incorrect facts and add missing ones. A final LLaMA2-13B-Chat extractor is trained on the denoised synthetic data to perform document-level relation triplet extraction. Across DocRED and Re-DocRED, GenRDK demonstrates substantial gains over strong baselines for both zero-shot document-level relation extraction and zero-shot triplet extraction, validating the effectiveness of retrieval-based data generation and cross-document denoising for reducing label noise in synthetic data.

Abstract

Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document. Existing methods heavily rely on a substantial amount of fully labeled data. However, collecting and annotating data for newly emerging relations is time-consuming and labor-intensive. Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities, inspiring us to explore an alternative approach for obtaining auto-labeled documents with new relations. In this paper, we propose a Zero-shot Document-level Relation Triplet Extraction (ZeroDocRTE) framework, which generates labeled data by retrieval and denoising knowledge from LLMs, called GenRDK. Specifically, we propose a chain-of-retrieval prompt to guide ChatGPT to generate labeled long-text data step by step. To improve the quality of synthetic data, we propose a denoising strategy based on the consistency of cross-document knowledge. Leveraging our denoised synthetic data, we proceed to fine-tune the LLaMA2-13B-Chat for extracting document-level relation triplets. We perform experiments for both zero-shot document-level relation and triplet extraction on two public datasets. The experimental results illustrate that our GenRDK framework outperforms strong baselines.

Figures (7)

• Figure 1: Comparison of sentence-level chia2022relationprompt and document-level data generated. In sentence-level synthetic data, there exists merely one relation triplet within a sentence. In the case of document-level synthetic data, there are more than 22 relation triplets distributed across different sentences. Entities and relations are marked in different colors.
• Figure 2: The original and denoised labels of a synthetic sample. Two main types of noise are reduced by our consistency-guided cross-document knowledge denoising strategy. One is reducing the incorrect triplet as shown in the red dotted line (The Godfather, Vito Corleone, cast member), and another is adding the missing triplet as shown in the green solid line (The Godfather, Francis Ford Coppola, screenwriter).
• Figure 3: The overview of our GenRDK framework. It contains four key parts as follows: (a) Chain-of-retrieval prompt for guiding ChatGPT to generate labeled data step by step; (b) Training the pre-denoising model based on LLaMA2-13B-Chat with LoRA; (c) Consistency-guided cross-document knowledge denoising strategy. (d) Training the relation triplet extractor with the denoised synthetic data.
• Figure 5: An example of the knowledge expressed by different generated documents. The relation triplets (The Godfather, Francis Ford Coppola, director) and (The Godfather, Mario Puzo, screenwriter) are multiply expressed in different synthetic documents.
• Figure 6: Illustration of vanilla and chain-of-thought prompt. Our chain-of-retrieval prompt can be seen in Figure \ref{['fig:4']}. We generate different groups of data by the above prompts.