FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

TL;DR

FedPT offers a promising solution for efficient, privacy-preserving fine-tuning of large LMs on resource-constrained devices, broadening the accessibility and applicability of state-of-the-art large LMs.

Abstract

Despite demonstrating superior performance across a variety of linguistic tasks, pre-trained large language models (LMs) often require fine-tuning on specific datasets to effectively address different downstream tasks. However, fine-tuning these LMs for downstream tasks necessitates collecting data from individuals, which raises significant privacy concerns. Federated learning (FL) has emerged as the de facto solution, enabling collaborative model training without sharing raw data. While promising, federated fine-tuning of large LMs faces significant challenges, including restricted access to model parameters and high computation, communication, and memory overhead. To address these challenges, this paper introduces \textbf{Fed}erated \textbf{P}roxy-\textbf{T}uning (FedPT), a novel framework for federated fine-tuning of black-box large LMs, requiring access only to their predictions over the output vocabulary instead of their parameters. Specifically, devices in FedPT first collaboratively tune a smaller LM, and then the server combines the knowledge learned by the tuned small LM with the knowledge learned by the larger pre-trained LM to construct a large proxy-tuned LM that can reach the performance of directly tuned large LMs. The experimental results demonstrate that FedPT can significantly reduce computation, communication, and memory overhead while maintaining competitive performance compared to directly federated fine-tuning of large LMs. FedPT offers a promising solution for efficient, privacy-preserving fine-tuning of large LMs on resource-constrained devices, broadening the accessibility and applicability of state-of-the-art large LMs.

Figures (19)

• Figure 1: Overview of FedPT. In FedPT, each training round comprises the following steps: 1) The cloud server broadcasts the latest small LM to the selected devices; 2) Each selected device fine-tunes the received small LM (e.g., using LoRA) and sends it back to the cloud server; 3) The cloud server collects and aggregates the updated small LMs; 4) The cloud server constructs the large proxy-tuned LM by utilizing the difference between the predictions of the small pre-trained and fine-tuned LMs to shift the original predictions of the larger pre-trained LM in the direction of tuning; and 5) The cloud server distills knowledge from the large proxy-tuned LM into the small aggregated LM to obtain the latest small LM. Note that FedPT does not require access to the internal model weight of the large pre-trained LM.
• Figure 2: Evaluation results of FedPT and baselines on LLaMA (a, b, c) and GPT-2 (d, e, f) models across different rounds for Dolly, SelfInst, and S-NI datasets. Higher Rouge-L scores indicate better performance.
• Figure 3: The scaling law of proxy-tuned models in the LLaMA family. FedAvg (7B) and FedAvg (30B) are directly fine-tuned models by FedAvg. At the 13B scale, we report the performance of FedPT (7B-13B) and FedAvg+PT (7B-13B). At the 30B scale, we use the fine-tuned 7B model from FedPT (7B-13B) to proxy-tune the 30B model for FedPT, and the 7B model from FedAvg (7B) to proxy-tune the 30B model for FedAvg+PT.
• Figure 4: Performance comparison of different $\alpha$ for FedPT on LLaMA across different rounds for Dolly, SelfInst, and S-NI datasets. Higher Rouge-L scores indicate better performance.
• Figure 5: Bar and pie charts of the number (a) and corresponding percentage (b) of each category in the Dolly dataset.