PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs

TL;DR

This work tackles the challenge of privacy-preserving on-device fine-tuning of large language models on resource-constrained smartphones by adopting derivative-free optimization, specifically the MeZo approach, to avoid storing gradients and optimizer states. It demonstrates that RoBERTa-large and OPT-1.3B can be fine-tuned on a commercial smartphone with memory footprints of about 4GB and 6.5GB, respectively, while traditional gradient-based methods fail due to out-of-memory. The results show that derivative-free fine-tuning can achieve meaningful loss reduction on-device, albeit with longer convergence times and significant hardware gaps compared to GPUs. The study highlights the feasibility of personalized, privacy-preserving LLMs on mobile devices and identifies practical directions for native deployment and hardware-aware optimizations to close the performance gap.

Abstract

Recent advancements in large language models (LLMs) have indeed showcased their impressive capabilities. On mobile devices, the wealth of valuable, non-public data generated daily holds great promise for locally fine-tuning personalized LLMs, while maintaining privacy through on-device processing. However, the constraints of mobile device resources pose challenges to direct on-device LLM fine-tuning, mainly due to the memory-intensive nature of derivative-based optimization required for saving gradients and optimizer states. To tackle this, we propose employing derivative-free optimization techniques to enable on-device fine-tuning of LLM, even on memory-limited mobile devices. Empirical results demonstrate that the RoBERTa-large model and OPT-1.3B can be fine-tuned locally on the OPPO Reno 6 smartphone using around 4GB and 6.5GB of memory respectively, using derivative-free optimization techniques. This highlights the feasibility of on-device LLM fine-tuning on mobile devices, paving the way for personalized LLMs on resource-constrained devices while safeguarding data privacy.

Figures (1)

• Figure 1: Training loss for fine-tuning RoBERTa-large using MeZo and Adam fine-tuning.