Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Graph-Assisted LLM Post-Training for Enhanced Legal Reasoning

Dezhao Song, Guglielmo Bonifazi, Frank Schilder, Jonathan Richard Schwarz

TL;DR

This work introduces an IRAC-based knowledge graph to encode legal reasoning and address the limitations of text-only post-training for high-stakes domains. By constructing an IRAC KG from 12K U.S. legal cases and generating SFT and DPO training data, the authors post-train mid-to-large LLMs (30B, 49B, 70B) and demonstrate improved performance across 5 diverse legal benchmarks, with the 70B DPO model achieving top results on multiple reasoning tasks. The study highlights the value of integrating structured domain knowledge into LLM training, showing gains over baselines and a SOTA 141B legal LLM, while also outlining limitations and future work for broader applicability and responsible deployment.

Abstract

LLM post-training has primarily relied on large text corpora and human feedback, without capturing the structure of domain knowledge. This has caused models to struggle dealing with complex reasoning tasks, especially for high-stakes professional domains. In Law, reasoning requires deep understanding of the relations between various legal concepts, a key component missing in current LLM post-training. In this paper, we propose a knowledge graph (KG)-assisted approach for enhancing LLMs' reasoning capability in Legal that is generalizable to other high-stakes domains. We model key legal concepts by following the \textbf{IRAC} (Issue, Rule, Analysis and Conclusion) framework, and construct a KG with 12K legal cases. We then produce training data using our IRAC KG, and conduct both Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) with three state-of-the-art (SOTA) LLMs (30B, 49B and 70B), varying architecture and base model family. Our post-trained models obtained better average performance on 4/5 diverse legal benchmarks (14 tasks) than baselines. In particular, our 70B DPO model achieved the best score on 4/6 reasoning tasks, among baselines and a 141B SOTA legal LLM, demonstrating the effectiveness of our KG for enhancing LLMs' legal reasoning capability.

Knowledge Graph-Assisted LLM Post-Training for Enhanced Legal Reasoning

TL;DR

This work introduces an IRAC-based knowledge graph to encode legal reasoning and address the limitations of text-only post-training for high-stakes domains. By constructing an IRAC KG from 12K U.S. legal cases and generating SFT and DPO training data, the authors post-train mid-to-large LLMs (30B, 49B, 70B) and demonstrate improved performance across 5 diverse legal benchmarks, with the 70B DPO model achieving top results on multiple reasoning tasks. The study highlights the value of integrating structured domain knowledge into LLM training, showing gains over baselines and a SOTA 141B legal LLM, while also outlining limitations and future work for broader applicability and responsible deployment.

Abstract

LLM post-training has primarily relied on large text corpora and human feedback, without capturing the structure of domain knowledge. This has caused models to struggle dealing with complex reasoning tasks, especially for high-stakes professional domains. In Law, reasoning requires deep understanding of the relations between various legal concepts, a key component missing in current LLM post-training. In this paper, we propose a knowledge graph (KG)-assisted approach for enhancing LLMs' reasoning capability in Legal that is generalizable to other high-stakes domains. We model key legal concepts by following the \textbf{IRAC} (Issue, Rule, Analysis and Conclusion) framework, and construct a KG with 12K legal cases. We then produce training data using our IRAC KG, and conduct both Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) with three state-of-the-art (SOTA) LLMs (30B, 49B and 70B), varying architecture and base model family. Our post-trained models obtained better average performance on 4/5 diverse legal benchmarks (14 tasks) than baselines. In particular, our 70B DPO model achieved the best score on 4/6 reasoning tasks, among baselines and a 141B SOTA legal LLM, demonstrating the effectiveness of our KG for enhancing LLMs' legal reasoning capability.
Paper Structure (26 sections, 9 figures, 7 tables, 2 algorithms)

This paper contains 26 sections, 9 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based KG Construction
  4. Graph-based LLM Reasoning
  5. KG-LLM Synergy in the Legal Domain
  6. System Overview and KG Design
  7. Overview
  8. Knowledge Graph Design
  9. Training Data Generation
  10. The Rules SFT Dataset
  11. The Rules Preference Dataset
  12. Model Training
  13. Evaluation
  14. IRAC KG Generation and Quality Review
  15. Evaluation Benchmarks
  16. ...and 11 more sections

Figures (9)

  • Figure 1: System overview.
  • Figure 2: Schema of our IRAC legal knowledge graph.
  • Figure 3: An example of the Rules SFT dataset.
  • Figure 4: An example of the Rules preference dataset.
  • Figure 5: Prompt for generating IRAC KG (Part I).
  • ...and 4 more figures