FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health

TL;DR

FedMentor tackles the privacy-utility trade-off in privately adapting LLMs for mental health by integrating domain-aware differential privacy with LoRA-based fine-tuning and FedAvg aggregation. The approach assigns per-domain privacy budgets and uses adaptive noise to maintain utility under non-IID data, achieving safer outputs while preserving near-centralized performance. Experiments on three mental-health datasets show safety improvements over non-private FL with minimal utility loss, and the method scales to backbones up to $1.7\times 10^9$ parameters with adapter-only communication on single-GPU clients. The work demonstrates a practical, privacy-preserving path for deploying safer mental-health LLMs in sensitive domains and beyond.

Abstract

Privacy-preserving adaptation of Large Language Models (LLMs) in sensitive domains (e.g., mental health) requires balancing strict confidentiality with model utility and safety. We propose FedMentor, a federated fine-tuning framework that integrates Low-Rank Adaptation (LoRA) and domain-aware Differential Privacy (DP) to meet per-domain privacy budgets while maintaining performance. Each client (domain) applies a custom DP noise scale proportional to its data sensitivity, and the server adaptively reduces noise when utility falls below a threshold. In experiments on three mental health datasets, we show that FedMentor improves safety over standard Federated Learning (FL) without privacy, raising safe output rates by up to three points and lowering toxicity, while maintaining utility (BERTScore F1 and ROUGE-L) within 0.5% of the non-private baseline and close to the centralized upper bound. The framework scales to backbones with up to 1.7B parameters on single-GPU clients, requiring < 173 MB of communication per-round. FedMentor demonstrates a practical approach to privately fine-tune LLMs for safer deployments in healthcare and other sensitive fields.

Figures (9)

• Figure 1: Comparison between traditional LLM-based methods for mental health and FedMentor. The baseline lacks privacy protection, robustness to heterogeneity, and communication efficiency, whereas FedMentor introduces domain-aware privacy, achieves robustness under non-IID data, and improves efficiency through LoRA-only updates.
• Figure 2: FedMentor pipeline. The server freezes the backbone and initializes LoRA adapters, layer scales, domain privacy budgets, and a utility threshold. Each round it broadcasts the current adapters; clients train LoRA on local data, add Gaussian noise per budget $\varepsilon_d$, and return noised adapters with a utility signal. The server aggregates with FedAvg and reduces noise when utility $<\tau$. After $R$ rounds the model is the frozen backbone plus learned LoRA adapters.
• Figure 3: FL
• Figure 4: FedMentor
• Figure 6: ParetoQ-350M: (a) REL and (b) B-F1 with TSR under Baseline, Static, and Uniform. Qwen3-1.7B: (c) $\varepsilon$ ablation of B-F1, TSR, and REL on global evaluation and on Dreaddit, IRF, and MultiWD. All panels report the final global model after 8 rounds (2 local epochs per-round).