CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correction

TL;DR

CooperLLM addresses the memory bottleneck of federated LLM fine-tuning on resource-constrained devices by marrying zeroth-order on-device updates with cloud-guided backpropagation to rectify gradient estimates. Its core, Zeroth-order Optimization with Gradient Rectification (ZGR), leverages a cloud-derived gradient subspace to generate guided perturbations that improve ZOO updates, while System-level Pipeline Controller (SPC) and Data Transmission Controller (DTC) overlap computation with communication and compress transmitted data to hide latency. Empirically, CooperLLM achieves up to 86% on-device memory reduction, up to 8.8× faster convergence, and up to 10 percentage-point accuracy gains across diverse Transformer models and datasets, outperforming state-of-the-art ZOO baselines. The approach demonstrates practical, privacy-preserving, and scalable FedLLM fine-tuning by combining algorithmic innovation with robust system-level optimizations across cloud-edge-end deployments.

Abstract

Large Language Models (LLMs) perform well on many NLP tasks, but fine-tuning them on resource-constrained mobile devices is challenging due to high memory and computation costs, despite growing demands for privacy-preserving personalization. Federated Learning (FL) enables local-data training, yet existing methods either rely on memory-intensive backpropagation or use zeroth-order optimization (ZOO), which avoids backward passes but suffers from slow convergence and degraded accuracy. We propose CooperLLM, a cloud-assisted edge-end cooperative federated fine-tuning framework that combines ZOO on mobile devices with cloud-guided gradient rectification. Mobile clients perform lightweight ZOO updates on private data, while the cloud fine-tunes on auxiliary public data using backpropagation and injects guided perturbations to rectify local updates, improving convergence and accuracy without violating privacy. To address system bottlenecks, CooperLLM introduces pipeline scheduling and adaptive compression to overlap computation and communication and reduce memory usage. Experiments on multiple Transformer models and datasets show that CooperLLM reduces on-device memory by up to $86.4\%$, accelerates convergence by $8.8 \times$, and improves accuracy by up to 10 percentage points over state-of-the-art ZOO-based baselines.

Figures (9)

• Figure 1: Memory breakdown between full fine-tuning and LoRA under varying prompt lengths with LlaMA-2-7B.
• Figure 2: The Comparison of BP-based Method and ZOO-based Method on Model Performance and Convergence Time with different Models on SST-2 stanfordnlp-sst2 Datasets.
• Figure 3: The Framework and Workflow of CooperLLM.
• Figure 4: The comparison of Without and With System-level Pipeline in CooperLLM.
• Figure 5: The Memory Footprint Comparison with/without SPC&DTC.