AgentRE: An Agent-Based Framework for Navigating Complex Information Landscapes in Relation Extraction

TL;DR

The paper tackles relation extraction in complex scenarios where diverse relation types and intra-sentential ambiguity challenge traditional text-in, text-out LMs. It introduces AgentRE, an agent-based framework that harnesses an LLM as a reasoning agent equipped with retrieval, memory, and extraction modules to perform multi-round, information-rich reasoning. Empirical results on English SciERC and Chinese DuIE show strong performance, especially under low-resource conditions, and ablations confirm the critical roles of retrieval and memory. Additionally, AgentRE generates informative reasoning trajectories that can distill into training data for smaller models, enabling cost-efficient deployment without sacrificing performance.

Abstract

The relation extraction (RE) in complex scenarios faces challenges such as diverse relation types and ambiguous relations between entities within a single sentence, leading to the poor performance of pure "text-in, text-out" language models (LMs). To address these challenges, in this paper, we propose an agent-based RE framework, namely AgentRE, which fully leverages the potential of large language models (LLMs) including memory, retrieval and reflection, to achieve RE in complex scenarios. Specifically, three major modules are built in AgentRE serving as the tools to help the agent acquire and process various useful information, thereby obtaining improved RE performance. Our extensive experimental results upon two datasets in English and Chinese demonstrate our AgentRE's superior performance, especially in low-resource scenarios. Additionally, the trajectories generated by AgentRE can be refined to construct a high-quality training dataset incorporating different reasoning methods, which can be used to fine-tune smaller models. Code is available at https://github.com/Lightblues/AgentRE.

Figures (4)

• Figure 1: Subfigure (a) illustrates the RE process of a language model of "text-in, text-out" scheme, which generates the results with errors directly from the input text or through simple prompting methods . Subfigure (b) illustrates the RE process of our proposed AgentRE, which is an agent-based framework including the retrieval and memory modules, and utilizes various information during multiple reasoning rounds to achieve more accurate RE.
• Figure 2: The overview of our proposed framework AgentRE. Subfigure (a) depicts the overall structure of AgentRE, where the LLM acts as an agent to extract relation triples from the input text through the collaboration with the retrieval, memory, and extraction module. Subfigures (b)$\sim$(d) illustrate the design of the retrieval, memory, and extraction Module, respectively.
• Figure 3: The prompt template for the retrieval module.
• Figure 4: AgentRE's performance on DuIE, including F1, Recall, and Precision with and without the memory module.