A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

TL;DR

This work addresses the challenge of efficiently fine-tuning large language models for many related and diverse tasks. It introduces CGC-LoRA, a unified framework that combines Customized Gate Control (CGC) with Low-Rank Adaptation (LoRA) to support a 1+N multi-task fine-tuning pattern, enabling a central LLM to serve multiple task clusters via task-common and task-specific experts. A task-motivated gate uses task IDs to determine expert contributions, with a fixed parameter budget comparable to standard LoRA, and training freezes the backbone while updating the CGC-LoRA modules. Empirical results on PromptCBLUE and Firefly show CGC-LoRA consistently outperforms strong baselines (including LoRA variants and MOE-LoRA) across 16 tasks, with ablations confirming the necessity of CGC and the gating mechanism. Collectively, this framework offers an efficient, scalable path to broad multi-task capability in LLMs, enabling improved performance with manageable compute and memory requirements.

Abstract

With the productive evolution of large language models (LLMs) in the field of natural language processing (NLP), tons of effort has been made to effectively fine-tune common pre-trained LLMs to fulfill a variety of tasks in one or multiple specific domain. In practice, there are two prevailing ways, in which the adaptation can be achieved: (i) Multiple Independent Models: Pre-trained LLMs are fine-tuned a few times independently using the corresponding training samples from each task. (ii) An Integrated Model: Samples from all tasks are employed to fine-tune a pre-trianed LLM unitedly. To address the high computing cost and seesawing issue simultaneously, we propose a unified framework that implements a 1 + N mutli-task fine-tuning pattern in LLMs using a novel Customized Gate Control (CGC) Low-rank Adaptation (LoRA) algorithm. Our work aims to take an advantage of both MTL (i.e., CGC) and PEFT (i.e., LoRA) scheme. For a given cluster of tasks, we design an innovative layer that contains two types of experts as additional trainable parameters to make LoRA be compatible with MTL. To comprehensively evaluate the proposed framework, we conduct well-designed experiments on two public datasets. The experimental results demonstrate that the unified framework with CGC-LoRA modules achieves higher evaluation scores than all benchmarks on both two datasets.

Figures (3)

• Figure 1: An overview of the proposed framework to implement $1+N$ multi-task fine-tuning pattern in LLMs. The first step is to divide a variety of tasks into $N$ clusters based on professional prior information or Inter-Task Affinity Fifty2021efficiently and the second step is to adapt a central LLM to each cluster of tasks using the LoRA fine-tuning algorithm of multi-task editions (e.g., MOE-LoRA, LoRAHub, and CGC-LoRA).
• Figure 2: A name entity recognition (NER) example to illustrate the whole pipeline of applications by large language models (LLMs). It includes the pre-processing to a conventional input, the textual answer of a LLM, the post-processing to extract final results.
• Figure 3: The detailed architecture of the proposed CGC-LoRA scheme. Specifically, the parameters of modules in blue are frozen while those of components in green, yellow, red are trainable during the fine-tuning process.