An LLM + ASP Workflow for Joint Entity-Relation Extraction
Trang Tran, Trung Hoang Le, Huiping Cao, Tran Cao Son
TL;DR
The paper tackles joint entity-relation extraction (JERE) under limited annotated data by pairing large language models (LLMs) with an Answer Set Programming (ASP) consistency checker. It introduces a modular, domain-agnostic prompt template and an ASP-based verifier that leverages optional domain type specifications to filter LLM outputs, reducing hallucinations. Empirical evaluation on CoNLL04, SciERC, and ADE demonstrates competitive or superior performance to state-of-the-art systems, with notable gains on SciERC even with only 10% of training data. The approach emphasizes flexibility and scalability, offering a pathway to robust JERE and potential knowledge-graph construction without heavy annotation or extensive fine-tuning.
Abstract
Joint entity-relation extraction (JERE) identifies both entities and their relationships simultaneously. Traditional machine-learning based approaches to performing this task require a large corpus of annotated data and lack the ability to easily incorporate domain specific information in the construction of the model. Therefore, creating a model for JERE is often labor intensive, time consuming, and elaboration intolerant. In this paper, we propose harnessing the capabilities of generative pre-trained large language models (LLMs) and the knowledge representation and reasoning capabilities of Answer Set Programming (ASP) to perform JERE. We present a generic workflow for JERE using LLMs and ASP. The workflow is generic in the sense that it can be applied for JERE in any domain. It takes advantage of LLM's capability in natural language understanding in that it works directly with unannotated text. It exploits the elaboration tolerant feature of ASP in that no modification of its core program is required when additional domain specific knowledge, in the form of type specifications, is found and needs to be used. We demonstrate the usefulness of the proposed workflow through experiments with limited training data on three well-known benchmarks for JERE. The results of our experiments show that the LLM + ASP workflow is better than state-of-the-art JERE systems in several categories with only 10% of training data. It is able to achieve a 2.5 times (35% over 15%) improvement in the Relation Extraction task for the SciERC corpus, one of the most difficult benchmarks.