Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Hua Ye, Siyuan Chen, Haoliang Zhang, Weihao Luo, Yanbin Li, Xuan Zhang
TL;DR
The paper tackles multi-domain fine-tuning of large language models by introducing a partition-based, multi-stage framework that clusters domains to maximize inter-domain synergy while controlling discrepancy and parameter budgets. The approach is supported by theoretical generalization bounds that incorporate domain discrepancy, synergy, and adapter/backbone capacity, and by an algorithm that efficiently partitions domains and performs stage-wise PEFT fine-tuning. Empirically, the method consistently exceeds baselines across four language-understanding tasks and multiple backbones, while reducing memory footprint and improving convergence. This synergy-aware partitioning offers a scalable, robust path for deploying LLMs across diverse domains, with strong implications for practical multi-domain adaptation and continual learning.
Abstract
Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.