Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs

Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, Fei Wu

TL;DR

AloRA is introduced, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one, and permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention.

Abstract

The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.

More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs

TL;DR

AloRA is introduced, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one, and permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention.

Abstract

The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.
Paper Structure (60 sections, 9 equations, 4 figures, 14 tables)

This paper contains 60 sections, 9 equations, 4 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs
  4. Catastrophic Forgetting
  5. General Capabilities Integration
  6. Task Formulation
  7. Tasks Design for Legal LLM
  8. Reasoning and Calculation
  9. Safe Dialogues
  10. Understanding to Legal Events
  11. Dataset Construction
  12. Article QA
  13. QA with Reasoning
  14. QA with Calculation
  15. Ethic
  16. ...and 45 more sections

Figures (4)

  • Figure 1: SFT on domain data injects domain knowledge into general LLMs. CF aims to keep the LLM performance on the general tasks after training on domain tasks. While GCI aims to enhance the performance on domain tasks by the integration of general capabilities with domain knowledge. Then the LLM is applied to domain-specific scenarios.
  • Figure 2: General Capabilities Integration enhances legal LLMs.
  • Figure 3: Framework of ALoRA.
  • Figure 4: Weights of the adapter outputs over tokens.