Continual Pre-Training is (not) What You Need in Domain Adaption

TL;DR

This work assesses Domain-Adaptive Continual Pre-Training (DACP) for legal AI within the Taiwanese-MMandarin legal system. It trains Llawa with domain-adaptive pre-training, followed by instruction tuning and preference alignment, and compares to LoRA-based baselines Blawstral and Bllawa across four legal reasoning tasks. Results show that while DACP enhances domain knowledge, it does not consistently improve prompting-based tasks or generalization, revealing trade-offs between domain specialization and flexible task performance. The study highlights the need for hybrid adaptation strategies and more nuanced benchmarks to reliably optimize legal reasoning in LLMs. Its findings inform practical design choices for domain-specific legal AI, including when and how to apply DACP versus alternative fine-tuning approaches.

Abstract

The recent advances in Legal Large Language Models (LLMs) have transformed the landscape of legal research and practice by automating tasks, enhancing research precision, and supporting complex decision-making processes. However, effectively adapting LLMs to the legal domain remains challenging due to the complexity of legal reasoning, the need for precise interpretation of specialized language, and the potential for hallucinations. This paper examines the efficacy of Domain-Adaptive Continual Pre-Training (DACP) in improving the legal reasoning capabilities of LLMs. Through a series of experiments on legal reasoning tasks within the Taiwanese legal framework, we demonstrate that while DACP enhances domain-specific knowledge, it does not uniformly improve performance across all legal tasks. We discuss the trade-offs involved in DACP, particularly its impact on model generalization and performance in prompt-based tasks, and propose directions for future research to optimize domain adaptation strategies in legal AI.

Figures (2)

• Figure 1: The correlation between model performance and question length for each of the grading criteria and the final grade.
• Figure 2: Model performance varies as question length increases for many of the models.