MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models

TL;DR

A method for fine-tuning Large Language Models, inspired by Multi-Task learning in a federated manner, utilizing a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), to reduce the number of trainable parameters.

Abstract

In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance.

Figures (3)

• Figure 1: A schematic illustration of the FMTL framework for fine-tuning LLMs. Each client is equipped with the same pre-trained model and a different task.
• Figure 2: Global model versus per-task model performance.
• Figure 3: Performance comparison of the proposed method and the baselines on Data-Juicer using the Natural Instruction dataset.