Fine-grained Contract NER using instruction based model

TL;DR

The paper tackles granular cause–effect extraction in financial texts by jointly evaluating supervised span-labeling and instruction-based prompting of LLMs. It introduces three prompt styles—GenPrompt, TaskPrompt, and CoTPrompt—to steer LLMs toward precise, generation-based extraction of cause and effect spans, while also leveraging LoRA-tuned sequence labeling models for efficiency. Empirical results show that CoTPrompt, particularly with ChatGPT, achieves the strongest exact-match performance among prompts and competitive ranking overall, with ablation analyses highlighting text overflow and label-swapping as key error modes. The work demonstrates that carefully designed prompts can enable robust information extraction in finance and points to future work in few-shot learning and prompt-tuning across additional LLMs to synergize LLM and supervised approaches.

Abstract

Lately, instruction-based techniques have made significant strides in improving performance in few-shot learning scenarios. They achieve this by bridging the gap between pre-trained language models and fine-tuning for specific downstream tasks. Despite these advancements, the performance of Large Language Models (LLMs) in information extraction tasks like Named Entity Recognition (NER), using prompts or instructions, still falls short of supervised baselines. The reason for this performance gap can be attributed to the fundamental disparity between NER and LLMs. NER is inherently a sequence labeling task, where the model must assign entity-type labels to individual tokens within a sentence. In contrast, LLMs are designed as a text generation task. This distinction between semantic labeling and text generation leads to subpar performance. In this paper, we transform the NER task into a text-generation task that can be readily adapted by LLMs. This involves enhancing source sentences with task-specific instructions and answer choices, allowing for the identification of entities and their types within natural language. We harness the strength of LLMs by integrating supervised learning within them. The goal of this combined strategy is to boost the performance of LLMs in extraction tasks like NER while simultaneously addressing hallucination issues often observed in LLM-generated content. A novel corpus Contract NER comprising seven frequently observed contract categories, encompassing named entities associated with 18 distinct legal entity types is released along with our baseline models. Our models and dataset are available to the community for future research * .