LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM

TL;DR

This paper tackles the gap in legal reasoning performance between proprietary and open-source LLMs by introducing KgDG, a knowledge-guided data generation framework that leverages a legal knowledge base to produce diverse, verifiable synthetic data. KgDG comprises three components—KgGen for generation, KgFix for refinement, and DaVer for verification—plus MiTra to expand the dataset and improve reasoning capabilities. The authors construct a 50K-question synthetic legal reasoning dataset and train LawGPT using mixtures of standard QA and reasoning-path QA, achieving performance surpassing many legal-specific LLMs and approaching proprietary-model levels at smaller scales. The work demonstrates practical potential for building capable open-source legal LLMs and provides publicly available code and resources to advance research in legal AI.

Abstract

Large language models (LLMs), both proprietary and open-source, have demonstrated remarkable capabilities across various natural language processing tasks. However, they face significant limitations in legal reasoning tasks. Proprietary models introduce data privacy risks and high inference costs, while open-source models underperform due to insufficient legal domain training data. To address these limitations, we study data generation for legal reasoning to improve the legal reasoning performance of open-source LLMs with the help of proprietary LLMs. This is challenging due to the lack of legal knowledge in proprietary LLMs and the difficulty in verifying the generated data. We propose KgDG, a knowledge-guided data generation framework for legal reasoning. Our framework enables leveraging legal knowledge to enhance generation diversity and introduces a refinement and verification process to ensure the quality of generated data. Moreover, we expand the generated dataset to further enhance the LLM reasoning capabilities. Using KgDG, we create a synthetic legal reasoning dataset containing 50K high-quality examples. Our trained model LawGPT outperforms existing legal-specific LLMs and achieves performance comparable to proprietary LLMs, demonstrating the effectiveness of KgDG and LawGPT. Our code and resources is publicly available at https://github.com/LAMDASZ-ML/Knowledge-Guide-Data-Generation .

Figures (2)

• Figure 1: Illustration of KgDG, a knowledge-guided data generation framework.
• Figure 2: Scalability analysis of the KgDG framework. The performance on all tasks improves as the amount of generated training data increases.