Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-driven Clustering and Merging of Adapters for On-device Large Language Models

Ondrej Bohdal, Taha Ceritli, Mete Ozay, Jijoong Moon, Kyeng-Hun Lee, Hyeonmok Ko, Umberto Michieli

TL;DR

This work tackles the challenge of deploying on-device LLMs under memory constraints by selecting a compact set of representative task adapters. It introduces D$^2$C, a data-driven clustering and merging algorithm that uses only a few task examples to iteratively refine cluster assignments and produce $K$ multi-task LoRAs from $N$ single-task LoRAs. Through extensive experiments across 40 text-generation tasks and multiple models, D$^2$C outperforms baselines while using only a fraction of the storage, enabling effective on-device functionality. The approach supports practical deployment by leveraging existing merging techniques (e.g., TIES) and demonstrating robustness across merging strategies and cluster configurations.

Abstract

On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobile devices typically have sufficient capacity to store a limited number of these parameters. This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks - a problem that remains unexplored in existing literature. We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples (e.g., 10 per task) and employs an iterative optimization process to refine cluster assignments. The adapters within each cluster are merged, creating multi-task adapters deployable on resource-constrained devices. Experimental results demonstrate that our method effectively boosts performance for considered storage budgets.

Data-driven Clustering and Merging of Adapters for On-device Large Language Models

TL;DR

This work tackles the challenge of deploying on-device LLMs under memory constraints by selecting a compact set of representative task adapters. It introduces DC, a data-driven clustering and merging algorithm that uses only a few task examples to iteratively refine cluster assignments and produce multi-task LoRAs from single-task LoRAs. Through extensive experiments across 40 text-generation tasks and multiple models, DC outperforms baselines while using only a fraction of the storage, enabling effective on-device functionality. The approach supports practical deployment by leveraging existing merging techniques (e.g., TIES) and demonstrating robustness across merging strategies and cluster configurations.

Abstract

On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobile devices typically have sufficient capacity to store a limited number of these parameters. This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks - a problem that remains unexplored in existing literature. We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples (e.g., 10 per task) and employs an iterative optimization process to refine cluster assignments. The adapters within each cluster are merged, creating multi-task adapters deployable on resource-constrained devices. Experimental results demonstrate that our method effectively boosts performance for considered storage budgets.
Paper Structure (8 sections, 4 figures, 1 table, 1 algorithm)

This paper contains 8 sections, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiments
  5. Setup
  6. Main Results
  7. Analyses
  8. Conclusion

Figures (4)

  • Figure 1: Overall pipeline. Single-task LoRAs are first clustered and then merged within the clusters. The resulting multi-task LoRAs are deployed to mobile devices under strict storage constraints.
  • Figure 2: Variable number of clusters. Our solution consistently improves performance and obtains stronger performance with more clusters. If each LoRA was in a separate cluster, there would be 40 clusters.
  • Figure 3: Clusters with variable homogeneity. Smaller $\alpha$ means each cluster is more homogeneous in terms of tasks / languages assigned to it. The results confirm the findings of our algorithm that it is better to group by task. In contrast, grouping by language does not work well.
  • Figure 4: Variable number of examples used by our algorithm. The performance does not vary significantly with the number of examples.