LAW: Legal Agentic Workflows for Custody and Fund Services Contracts

TL;DR

The paper tackles the challenge of extracting and reasoning over lengthy, dense custody and fund services contracts where standard LLMs struggle with context limits and legal jargon. It proposes LAW, a modular, agentic framework that orchestrates domain-specific tools and text agents via a code-generation orchestrator, enabling multi-hop reasoning and reusable components. Empirical results show LAW significantly outperforms a GPT-3.5-turbo baseline on retrieval and analytical tasks, most notably a 92.9 percentage-point gain on multi-hop termination-date calculations, while also reducing hallucinations and enabling scalable extraction across thousands of documents. The approach offers a cost-effective alternative to fine-tuned legal LLMs and demonstrates practical potential for automated, end-to-end workflows in custody and fund services contracts, with future work extending to non-English contracts and other domains.

Abstract

Legal contracts in the custody and fund services domain govern critical aspects such as key provider responsibilities, fee schedules, and indemnification rights. However, it is challenging for an off-the-shelf Large Language Model (LLM) to ingest these contracts due to the lengthy unstructured streams of text, limited LLM context windows, and complex legal jargon. To address these challenges, we introduce LAW (Legal Agentic Workflows for Custody and Fund Services Contracts). LAW features a modular design that responds to user queries by orchestrating a suite of domain-specific tools and text agents. Our experiments demonstrate that LAW, by integrating multiple specialized agents and tools, significantly outperforms the baseline. LAW excels particularly in complex tasks such as calculating a contract's termination date, surpassing the baseline by 92.9% points. Furthermore, LAW offers a cost-effective alternative to traditional fine-tuned legal LLMs by leveraging reusable, domain-specific tools.

Figures (2)

• Figure 1: Comparing the traditional method of fine-tuning a legal LLM vs. LAW (Legal Agentic Workflows). Fine-tuning involves labeling contracts for a highly customized pipeline supported by open-source LLMs, which results in limited context, scale, or flexibility where coaxing additional information out of the model would require further tuning. The ensuing prompt engineering on the fine-tuned legal LLM also exacerbates the model's loss of generality. In contrast, LAW can operate on both closed-source or open-source LLMs and is equipped with legal domain-specific tools. These tools are cheaper to construct, reusable, simpler to construct, and incorporates recent data. The general LLM's orchestration of these tools along with the text agents engenders LAW, a highly interactive agentic system that also enables the addition of more tools and agents.
• Figure 2: System Overview of LAW. (1) User query input on front-end (§\ref{['subsec:infra_question_templates']}); (2) Query manipulation and custom modification added to prompt and sent to the code generation agent (§\ref{['subsec:code_genagent']}); (3) Chat completion return from the code generation agent; (4) Execution of backend API tools (§\ref{['sec:tools']}); (5) Tool retrieval of information from internal cache (§\ref{['subsec:infra_caching']}); (6) Calls to text agents (§\ref{['sec:text_agents']}); (7) Calls to multi-node OpenSearch cluster for text retrieval (§\ref{['subsec:infra_opensearch']}); (8) Feedback on code runs back to the code generation agent in case of failure; (9) Concatenation of final output; (10) Final text output is rendered on the UI; (11) EDGAR contracts undergo continuous, offline, distributed processing to update our internal cache and OpenSearch systems (§\ref{['sec:data_source']}).