Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, Michael Desa

TL;DR

SaulLM-54B and SaulLM-141B address the challenge of domain adaptation for the legal domain by scaling model size and corpus and by a holistic adaptation pipeline. The authors train $54$B and $141$B Mixture-of-Experts models on a base legal corpus exceeding $500$B tokens, then apply instruction fine-tuning with legal data and preference alignment using synthetic data, releasing base, instruct, and aligned variants under MIT. They report state-of-the-art performance on LegalBench-Instruct and strong performance on Legal-MMLU compared with GPT-4, Llama3, and Mixtral baselines, while highlighting the value of continued pretraining, legal data, and DPO. The work also analyzes scalability and energy consumption, and discusses limitations related to reproducibility and reliance on proprietary datasets for alignment, offering insights for future domain-adaptation research.

Abstract

In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language models (LLMs) tailored for the legal sector. These models, which feature architectures of 54 billion and 141 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is guided by large-scale domain adaptation, divided into three strategies: (1) the exploitation of continued pretraining involving a base corpus that includes over 540 billion of legal tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming previous open-source models on LegalBench-Instruct. This work explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks. We are releasing base, instruct, and aligned versions on top of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and collaborative research.

SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

TL;DR

SaulLM-54B and SaulLM-141B address the challenge of domain adaptation for the legal domain by scaling model size and corpus and by a holistic adaptation pipeline. The authors train B and B Mixture-of-Experts models on a base legal corpus exceeding B tokens, then apply instruction fine-tuning with legal data and preference alignment using synthetic data, releasing base, instruct, and aligned variants under MIT. They report state-of-the-art performance on LegalBench-Instruct and strong performance on Legal-MMLU compared with GPT-4, Llama3, and Mixtral baselines, while highlighting the value of continued pretraining, legal data, and DPO. The work also analyzes scalability and energy consumption, and discusses limitations related to reproducibility and reliance on proprietary datasets for alignment, offering insights for future domain-adaptation research.

Abstract

In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language models (LLMs) tailored for the legal sector. These models, which feature architectures of 54 billion and 141 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is guided by large-scale domain adaptation, divided into three strategies: (1) the exploitation of continued pretraining involving a base corpus that includes over 540 billion of legal tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming previous open-source models on LegalBench-Instruct. This work explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks. We are releasing base, instruct, and aligned versions on top of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and collaborative research.
Paper Structure (28 sections, 18 figures, 3 tables)

This paper contains 28 sections, 18 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Domain Specialization For Large Language Models
  4. Legal Domain Adaptation for Modern LLM
  5. Data Collection and Corpus Construction
  6. Pretraining Corpora
  7. Legal Sources
  8. Other Sources
  9. Data Preprocessing
  10. Instruction Data
  11. Preference Data
  12. Implementation Details & Evaluation Protocol
  13. Model Selection
  14. Engineering Details
  15. Training Details
  16. ...and 13 more sections

Figures (18)

  • Figure 1: Sources of Legal Pretraining Data
  • Figure 1: Domain adaptation method model for turning a Mixtral to a SaulLM-141B. Training involves different stages: legal domain pretraining, instruction filtering, and preference filtering.
  • Figure 2: Overall Results. Comparison of SaulLM-large and SaulLM-medium with existing models.
  • Figure 3: Jurisprudence
  • Figure 4: Professional Law
  • ...and 13 more figures