Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges

TL;DR

This paper evaluates the ability of ChatGPT (GPT-3.5-turbo) to extract normative relationships from contracts, focusing on four norm types—commitment, prohibition, authorization, and power—and their elements (subject, object, antecedent, consequent). Using the CUAD dataset, the authors craft prompts and perform a qualitative analysis on 100 clauses plus 50 challenging ones to assess zero-shot norm extraction without annotated data. The findings show that ChatGPT can extract norms in many cases but suffers from several reliability issues, including incorrect norm types, incorrect elements, omission of crucial details, hallucinations, and poor handling of conjunctions, with redacted clauses exacerbating errors. The study highlights the scarcity and quality problems of existing contract datasets and outlines directions for improving prompt design, conjunction handling, and data quality to enable more trustworthy norm specifications for multiagent systems. Overall, the work demonstrates both the promise and current limits of using LLMs for automated contract understanding and norm extraction, outlining concrete paths for future research and practical impact in governance of autonomous agents.

Abstract

We investigate the effectiveness of ChatGPT in extracting norms from contracts. Norms provide a natural way to engineer multiagent systems by capturing how to govern the interactions between two or more autonomous parties. We extract norms of commitment, prohibition, authorization, and power, along with associated norm elements (the parties involved, antecedents, and consequents) from contracts. Our investigation reveals ChatGPT's effectiveness and limitations in norm extraction from contracts. ChatGPT demonstrates promising performance in norm extraction without requiring training or fine-tuning, thus obviating the need for annotated data, which is not generally available in this domain. However, we found some limitations of ChatGPT in extracting these norms that lead to incorrect norm extractions. The limitations include oversight of crucial details, hallucination, incorrect parsing of conjunctions, and empty norm elements. Enhanced norm extraction from contracts can foster the development of more transparent and trustworthy formal agent interaction specifications, thereby contributing to the improvement of multiagent systems.