Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models

TL;DR

This work tackles catastrophic forgetting and inter-domain confusion when adapting LLMs to multiple domains. It introduces REGA, a three-component framework—Self-Distillation, Role Prompting, and Role Integration—with a central prompt guiding inference to achieve strong domain performance while preserving broad capabilities. Empirical results across English and Chinese medicine, law, and finance datasets show REGA outperforms standard finetuning and role-prompt baselines, with ablations confirming the value of each component. The approach offers a scalable, deployable solution for multi-domain LLM adaptation that maintains robust generality useful for real-world, instruction-following systems.

Abstract

The growing interest in Large Language Models (LLMs) for specialized applications has revealed a significant challenge: when tailored to specific domains, LLMs tend to experience catastrophic forgetting, compromising their general capabilities and leading to a suboptimal user experience. Additionally, crafting a versatile model for multiple domains simultaneously often results in a decline in overall performance due to confusion between domains. In response to these issues, we present the RolE Prompting Guided Multi-Domain Adaptation (REGA) strategy. This novel approach effectively manages multi-domain LLM adaptation through three key components: 1) Self-Distillation constructs and replays general-domain exemplars to alleviate catastrophic forgetting. 2) Role Prompting assigns a central prompt to the general domain and a unique role prompt to each specific domain to minimize inter-domain confusion during training. 3) Role Integration reuses and integrates a small portion of domain-specific data to the general-domain data, which are trained under the guidance of the central prompt. The central prompt is used for a streamlined inference process, removing the necessity to switch prompts for different domains. Empirical results demonstrate that REGA effectively alleviates catastrophic forgetting and inter-domain confusion. This leads to improved domain-specific performance compared to standard fine-tuned models, while still preserving robust general capabilities.

Figures (5)

• Figure 1: Performance Comparison of BELLE with varied sizes tuned by Standard Finetuning (FT) and REGA. The models tuned by FT suffer from a severe drop in general performance as the training epoch increases. Whereas, the counterparts tuned by REGA are better at preserving general capacities while achieving comparable domain-specific performance.
• Figure 2: Overview of REGA. For training, REGA organizes the training data by: (1) Self-Distillation: The vanilla LLM generates exemplars according to a set of general-domain instructions to preserve generic abilities. (2) Role Prompting: The LLM is assigned a unique role through role prompts, which are concatenated with samples in corresponding domains. $P_C$ is the central prompt indicating the generalist role for the general domain, while $P_L$, $P_M$, and $P_F$ are the expert role prompts for law, medicine, and finance domains. (3) Role Integration: A fraction of data from each specialized domain is mixed with the general-domain data, all guided by the central prompt, which integrates various expert roles into the generalist role. For inference, the central prompt effectively guides the LLM tuned on REGA training data to respond to multi-domain instructions, without the need for role prompt selection.
• Figure 3: Performance of BELLE and Vicuna tuned by FTRP. FTRP$^x$ indicate models are tested by the role prompts $p_x$ of the $x$ domain.
• Figure 4: We present the normalized performance metrics for the BELLE and Vicuna-7B, which are fine-tuned using the FTSD and REGA. The notation REGA$^x$ indicates that the model's inference is performed using the role prompt $p_x$. The normalization process involves dividing each score by the maximum score within the same column. The mixing ratio of REGA is 0.1.
• Figure 5: General and domain performance of BELLE-7B and Vicuna-7B trained with a varied mixing ratio of REGA.