Data Augmented Pipeline for Legal Information Extraction and Reasoning

TL;DR

The paper tackles the data bottleneck in adapting a legal information-extraction system (Deep PROLEG) to new contracts by introducing a data-augmentation pipeline that leverages few-shot prompting with LLMs to generate domain-specific templates, slot holders, and PROLEG facts. It evaluates two neural semantic parsing strategies (end-to-end and NER-based) trained on augmented data to enable rapid domain adaptation. Empirical results show ChatGPT-based augmentation achieving over 95% accuracy on 5,000 augmented cases across multiple contract types and slot holders, outperforming some open-source LLMs. The work claims broad applicability to other NLP tasks beyond the legal domain and offers a practical framework for reducing annotation effort in information-extraction systems.

Abstract

In this paper, we propose a pipeline leveraging Large Language Models (LLMs) for data augmentation in Information Extraction tasks within the legal domain. The proposed method is both simple and effective, significantly reducing the manual effort required for data annotation while enhancing the robustness of Information Extraction systems. Furthermore, the method is generalizable, making it applicable to various Natural Language Processing (NLP) tasks beyond the legal domain.

Figures (1)

• Figure 1: The general architecture of the deep PROLEG system is depicted, along with the distribution of legal slot holders in the augmented data. The components related to the process of adding a new domain are highlighted in red.