FedJudge: Federated Legal Large Language Model
Linan Yue, Qi Liu, Yichao Du, Weibo Gao, Ye Liu, Fangzhou Yao
TL;DR
This work tackles the privacy problem in training legal large language models by introducing FedJudge, a federated learning framework that fine-tunes base LLMs in a privacy-preserving, data-heterogeneity-aware manner. It leverages parameter-efficient fine-tuning via LoRA (updating only the trainable $\mathcal{W}^e$) and aggregates these updates across clients to form a global $\hat{\mathcal{W}}^e$, while freezing the large backbone $\mathcal{W}^p$ to reduce compute and communication costs. To mitigate distribution shifts across legal domains, FedJudge extends to FedJudge-CL, which imposes a continual learning constraint based on parameter importance $J^{(t-1)}$ to preserve global knowledge during local updates, formalized as $\mathcal{L}_i^{* (t)} = \mathcal{L}_i^{(t)} + \mathbb{I}(t \neq 1) \times \lambda \mathcal{L}_i^{cl (t)}$. Empirical results on three real-world legal tasks (court views, legal consultations, and legal reasoning) show that FedJudge-CL achieves robust performance improvements over both centralized and standard FL baselines, with favorable human evaluations, demonstrating the framework’s practicality for privacy-preserving, capable legal AI.
Abstract
Large Language Models (LLMs) have gained prominence in the field of Legal Intelligence, offering potential applications in assisting legal professionals and laymen. However, the centralized training of these Legal LLMs raises data privacy concerns, as legal data is distributed among various institutions containing sensitive individual information. This paper addresses this challenge by exploring the integration of Legal LLMs with Federated Learning (FL) methodologies. By employing FL, Legal LLMs can be fine-tuned locally on devices or clients, and their parameters are aggregated and distributed on a central server, ensuring data privacy without directly sharing raw data. However, computation and communication overheads hinder the full fine-tuning of LLMs under the FL setting. Moreover, the distribution shift of legal data reduces the effectiveness of FL methods. To this end, in this paper, we propose the first Federated Legal Large Language Model (FedJudge) framework, which fine-tunes Legal LLMs efficiently and effectively. Specifically, FedJudge utilizes parameter-efficient fine-tuning methods to update only a few additional parameters during the FL training. Besides, we explore the continual learning methods to preserve the global model's important parameters when training local clients to mitigate the problem of data shifts. Extensive experimental results on three real-world datasets clearly validate the effectiveness of FedJudge. Code is released at https://github.com/yuelinan/FedJudge.