Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

Donald Shenaj, Ondrej Bohdal, Taha Ceritli, Mete Ozay, Pietro Zanuttigh, Umberto Michieli

TL;DR

The paper tackles on-device continual adaptation of LLMs by merging LoRA adapters under a fixed storage budget. It introduces data-free, similarity-based clustering and history-aware merging, implemented in two variants, K-Merge and K-Merge++, to efficiently integrate new adapters while preserving performance on previously seen tasks. Across 40 LoRAs, five problem types, and eight languages, the approach outperforms static baselines and other merging methods, achieving substantial task coverage with minimal storage and computation. This work enables private, scalable on-device personalization for multilingual, multi-task LLMs without requiring access to training data.

Abstract

On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile devices, recent works have explored model merging techniques to fuse multiple LoRAs into a single one. In practice, however, LoRAs are often delivered incrementally, as users request support for new tasks (e.g., novel problem types or languages). This scenario introduces a new challenge: on-device online continual merging, where the objective is to incorporate new LoRAs while preserving the performance on previously supported tasks. In this paper, we propose a data-free and computationally efficient strategy for selecting and merging LoRAs when a new one becomes available, assuming the device can store only a limited number of adapters. Extensive experiments across real-world tasks demonstrate the superiority of our approach compared to alternative strategies while adhering to the storage budget and compute limitations of on-device settings.

K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

TL;DR

The paper tackles on-device continual adaptation of LLMs by merging LoRA adapters under a fixed storage budget. It introduces data-free, similarity-based clustering and history-aware merging, implemented in two variants, K-Merge and K-Merge++, to efficiently integrate new adapters while preserving performance on previously seen tasks. Across 40 LoRAs, five problem types, and eight languages, the approach outperforms static baselines and other merging methods, achieving substantial task coverage with minimal storage and computation. This work enables private, scalable on-device personalization for multilingual, multi-task LLMs without requiring access to training data.

Abstract

On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile devices, recent works have explored model merging techniques to fuse multiple LoRAs into a single one. In practice, however, LoRAs are often delivered incrementally, as users request support for new tasks (e.g., novel problem types or languages). This scenario introduces a new challenge: on-device online continual merging, where the objective is to incorporate new LoRAs while preserving the performance on previously supported tasks. In this paper, we propose a data-free and computationally efficient strategy for selecting and merging LoRAs when a new one becomes available, assuming the device can store only a limited number of adapters. Extensive experiments across real-world tasks demonstrate the superiority of our approach compared to alternative strategies while adhering to the storage budget and compute limitations of on-device settings.
Paper Structure (27 sections, 4 equations, 8 figures, 9 tables, 1 algorithm)

This paper contains 27 sections, 4 equations, 8 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Online Continual Merging
  5. Evaluation Protocol and Aggregate Score
  6. Proposed Method
  7. Cluster Assignment by Similarity Scoring
  8. Iterative History-aware Model Merging
  9. K-Merge and K-Merge$++$
  10. Inference Stage
  11. Experimental Evaluation
  12. Experimental Details
  13. Experimental Results
  14. Conclusion
  15. Additional Details
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Online continual merging of adapters for on-device LLMs. Each adapter corresponds to a specific task $\tau_t$ (e.g., a specific problem type in a selected language). The objective is to increase the LLM capabilities over time without storing all adapters, but rather using a budget of $K$ adapters (e.g., $K\!=\!4$ here). The key steps are: 1) The device downloads new adapters over time. 2) The system selects the most similar stored adapter to the new one. 3) The system updates the selected stored adapter by merging it with the new adapter.
  • Figure 2: K-Merge$++$ outline. When the storage budget is not fulfilled and a new LoRA $L^{(t)}$ is downloaded, it is compared with the LoRAs in $\mathcal{L}^{(t-1)}$ using Eq. \ref{['eq:sim']}. If similarity is smaller than the threshold $s$, $L^{(t)}$ initializes a new cluster; otherwise, it gets merged with the closest LoRA following Eq. \ref{['eq:closest']}. After the budget limit has been reached, new LoRAs can only be merged.
  • Figure 3: Score $S^{(\gamma)}$ of compared methods at variable storage budget $K$. Averaged over 3 random task orderings.
  • Figure 4: Performance $S^{(t)}$ on discovered tasks at varying timesteps for $K=5$ using Llama-3.2-1B (left) and Qwen-2.5-1.5B (right).
  • Figure A.1: Analysis of the time it takes to integrate a new LoRA for the K-Merge approach (values for K-Merge$++$ are roughly the same). The time is proportional to the number of stored LoRAs (clusters).
  • ...and 3 more figures