Federated Co-tuning Framework for Large and Small Language Models

TL;DR

FedCoLLM tackles the challenge of domain-specific adaptation under privacy and resource constraints by enabling mutual enhancement between a server-side LLM and client-side SLMs. The method hinges on parameter-efficient fine-tuning via LoRA adapters and mutual knowledge distillation, orchestrated through secure aggregation to keep data private. Empirical results across multiple LLM/SLM pairings and QA tasks show that client SLMs gain notable improvements from the server’s knowledge, while the server LLM approaches the performance of centrally fine-tuned servers, all at significantly reduced communication costs. The work demonstrates a practical pathway for scalable, privacy-preserving federated co-tuning with strong implications for deploying domain-specific LLM ecosystems.

Abstract

By adapting Large Language Models (LLMs) to domain-specific tasks or enriching them with domain-specific knowledge, we can fully harness the capabilities of LLMs. Nonetheless, a gap persists in achieving simultaneous mutual enhancement between the server's LLM and the downstream clients' Small Language Models (SLMs). To address this, we propose FedCoLLM, a novel and parameter-efficient federated framework designed for co-tuning LLMs and SLMs. This approach is aimed at adaptively transferring server-side LLMs knowledge to clients' SLMs while simultaneously enriching the LLMs with domain insights from the clients. To accomplish this, FedCoLLM utilizes lightweight adapters in conjunction with SLMs, facilitating knowledge exchange between server and clients in a manner that respects data privacy while also minimizing computational and communication overhead. Our evaluation of FedCoLLM, utilizing various public LLMs and SLMs across a range of NLP text generation tasks, reveals that the performance of clients' SLMs experiences notable improvements with the assistance of the LLMs. Simultaneously, the LLMs enhanced via FedCoLLM achieves comparable performance to that obtained through direct fine-tuning on clients' data. Our code has been contributed to the FATE open-source project and is now publicly accessible at https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/fedcollm.

Figures (1)

• Figure 1: FedCoLLM (Federated parameter-efficient co-tuning of clients’ domain SLMs and the server’s LLMs. Clients’ SLMs learn from each other via federated fine-tuning of their adapter modules and transfer knowledge from and to the server’s LLM)