Knowledge Restoration-driven Prompt Optimization: Unlocking LLM Potential for Open-Domain Relational Triplet Extraction

TL;DR

The paper addresses the bottlenecks of open-domain relational triplet extraction by introducing KRPO, a knowledge restoration–driven framework that continually refines prompts through a self-evaluation loop and textual gradients. It also incorporates a dynamic relation canonicalization memory powered by a cross-encoder to align and standardize relational schemas, reducing semantic redundancy. Empirical results across three datasets and multiple LLM backbones show that KRPO consistently outperforms strong baselines, with notable gains in high-fidelity, strictly correct triplets, especially for smaller models. This work advances open-domain KG construction by enabling iterative prompt refinement and robust relation normalization, enabling more reliable knowledge extraction in diverse linguistic contexts.

Abstract

Open-domain Relational Triplet Extraction (ORTE) is the foundation for mining structured knowledge without predefined schemas. Despite the impressive in-context learning capabilities of Large Language Models (LLMs), existing methods are hindered by their reliance on static, heuristic-driven prompting strategies. Due to the lack of reflection mechanisms required to internalize erroneous signals, these methods exhibit vulnerability in semantic ambiguity, often making erroneous extraction patterns permanent. To address this bottleneck, we propose a Knowledge Reconstruction-driven Prompt Optimization (KRPO) framework to assist LLMs in continuously improving their extraction capabilities for complex ORTE task flows. Specifically, we design a self-evaluation mechanism based on knowledge restoration, which provides intrinsic feedback signals by projecting structured triplets into semantic consistency scores. Subsequently, we propose a prompt optimizer based on a textual gradient that can internalize historical experiences to iteratively optimize prompts, which can better guide LLMs to handle subsequent extraction tasks. Furthermore, to alleviate relation redundancy, we design a relation canonicalization memory that collects representative relations and provides semantically distinct schemas for the triplets. Extensive experiments across three datasets show that KRPO significantly outperforms strong baselines in the extraction F1 score.

Figures (6)

• Figure 1: Comparison between KRPO (Ours) and EDC EDC. KRPO dynamically optimizes prompts via self-reflection, releasing LLM's potential for ORTE, whereas EDC relies on static prompts and suffers from performance stagnation.
• Figure 2: Overview of KRPO. Comprises four modules: (1) Relational Triplet Extraction (RTE), extracting triplets by LLM with an optimizable prompt; (2) Self Evaluation, assessing consistency via NLI on restored text; (3) Prompt Optimization, optimizing the prompt using evaluation feedback as gradients; and (4) Relation Canonicalization, aligning relations with dynamic schemas for Knowledge Graph storage.
• Figure 3: Ablation study on WebNLG, REBEL, and Wiki-NRE. We report the Strict-F1 scores to analyze the contribution of different components (e.g., PO and RC modules) to the final performance.
• Figure 4: Analysis of Relation Canonicalization Accuracy under different $k$. Evaluated on triplets with correct "(subject,object)" pair but incorrect "relation".
• Figure 5: A case study comparing the extraction results between the Initial and Updated prompts. The optimized prompt (right) incorporates more detailed constraints (in blue), guiding the LLM to generate triplets that are more precise compared to the initial prompt (left).