Efficient Compositional Multi-tasking for On-device Large Language Models

TL;DR

This work tackles compositional multi-tasking for on-device LLMs, where a single input must be processed by multiple tasks concurrently. It shows that existing merging and baselines struggle to perform such tasks efficiently, and introduces Learnable Calibration, a lightweight calibration mechanism that builds on already-available task-specific LoRAs to solve compositional tasks in a single pass. A new benchmark with four practical compositional task configurations and fourteen sub-tasks demonstrates the method’s effectiveness, achieving strong performance with minimal additional storage and computation. The approach promises practical on-device capabilities for complex multi-tasking applications, while motivating further research into scalability, domain transfer, and safeguards. The work includes extensive experiments across multiple model sizes and languages, highlighting the robustness and efficiency of Learnable Calibration, particularly its variant LC++.

Abstract

Adapter parameters provide a mechanism to modify the behavior of machine learning models and have gained significant popularity in the context of large language models (LLMs) and generative AI. These parameters can be merged to support multiple tasks via a process known as task merging. However, prior work on merging in LLMs, particularly in natural language processing, has been limited to scenarios where each test example addresses only a single task. In this paper, we focus on on-device settings and study the problem of text-based compositional multi-tasking, where each test example involves the simultaneous execution of multiple tasks. For instance, generating a translated summary of a long text requires solving both translation and summarization tasks concurrently. To facilitate research in this setting, we propose a benchmark comprising four practically relevant compositional tasks. We also present an efficient method (Learnable Calibration) tailored for on-device applications, where computational resources are limited, emphasizing the need for solutions that are both resource-efficient and high-performing. Our contributions lay the groundwork for advancing the capabilities of LLMs in real-world multi-tasking scenarios, expanding their applicability to complex, resource-constrained use cases.

Figures (17)

• Figure 1: Compositional multi-tasking involves performing multiple tasks simultaneously, such as summarization and translation. The challenge lies in executing all tasks jointly within a single inference pass for optimal efficiency, rather than performing them separately through multiple inferences.
• Figure 2: Overview of the four compositional tasks in our benchmark. The tasks include three translation settings (English to Spanish, French, and German) and four tone variations (professional, casual, witty, and neutral paraphrase), leading to fourteen sub-tasks overall.
• Figure 3: LoRA weight update matrices differ across tasks, so sharing additional parameters $P$ is likely to be suboptimal.
• Figure 4: Our Learnable Calibration. We add a small number of calibration parameters to correct the initial merged LoRAs. Variation #1 uses a calibration vector of biases, while Variation #2 (++) uses two calibration low-rank matrices.
• Figure 5: Changes in the update matrix. Learnable Calibration makes the overall LoRA update matrix significantly more diverse to handle additional tasks. The weight norms also increase substantially across different components, reflecting the added complexity required for compositional multi-tasking.