Equitable Access to Justice: Logical LLMs Show Promise

TL;DR

This paper explores the integration of LLMs with logic programming to enhance their ability to reason, bringing their strategic capabilities closer to that of a skilled lawyer and demonstrates how LLMs with advanced System 2 reasoning capabilities can expand access to justice.

Abstract

The costs and complexity of the American judicial system limit access to legal solutions for many Americans. Large language models (LLMs) hold great potential to improve access to justice. However, a major challenge in applying AI and LLMs in legal contexts, where consistency and reliability are crucial, is the need for System 2 reasoning. In this paper, we explore the integration of LLMs with logic programming to enhance their ability to reason, bringing their strategic capabilities closer to that of a skilled lawyer. Our objective is to translate laws and contracts into logic programs that can be applied to specific legal cases, with a focus on insurance contracts. We demonstrate that while GPT-4o fails to encode a simple health insurance contract into logical code, the recently released OpenAI o1-preview model succeeds, exemplifying how LLMs with advanced System 2 reasoning capabilities can expand access to justice.

Figures (2)

• Figure 1: Prolog translation of simplified Chubb policy generated by GPT-4o. Corresponding code in Appendix \ref{['app:4o_policy_encoding']}
• Figure 2: Prolog translation of simplified Chubb policy generated by OpenAI o1-preview. Corresponding code in Appendix \ref{['app:o1_policy_encoding']}