SynLexLM: Scaling Legal LLMs with Synthetic Data and Curriculum Learning

TL;DR

This work introduces SynLexLM, a novel approach to efficiently pre-train a legal LLM, and aims to achieve improved performance on legal benchmarks compared to traditional models and fine-tuned versions.

Abstract

Large Language Models (LLMs) are powerful but often require extensive fine-tuning and large datasets for specialized domains like law. General-purpose pre-training may not capture legal nuances, and acquiring sufficient legal data is challenging. We introduce SynLexLM, a novel approach to efficiently pre-train a legal LLM. Our method employs curriculum learning, progressing from simple to complex legal texts and queries, combined with synthetic data augmentation using models like Gemini Pro to address data scarcity. We aim to achieve improved performance on legal benchmarks (BigLaw-Bench, EUR-Lex-Sum) compared to traditional models and fine-tuned versions. Preliminary work involves generating synthetic QA pairs reflecting legal reasoning. This work aims to enhance legal document analysis and research tools, potentially democratizing access to advanced legal AI.

Figures (3)

• Figure 1: Architecture diagram of the SynLexLM system, illustrating the flow from real data curation through synthetic data generation, curriculum learning, model fine-tuning, and evaluation.
• Figure 3: EurLex-Sum Training Losses Plot
• Figure 4: EurLex Training Losses Plot