On-device System of Compositional Multi-tasking in Large Language Models

TL;DR

The paper tackles enabling compositional multi-tasking (summarization plus translation) entirely on-device to preserve privacy and reduce latency. It introduces a projection-merge method that adds a small learnable projection on top of pre-trained LoRA adapters to fuse two single-task capabilities into a single inference pass. Results show the projection merge matches or surpasses inefficient baselines like two-step LoRA and joint-expert while using far fewer extra parameters and enabling fully on-device operation. This work demonstrates a practical, privacy-preserving, fast on-device solution and outlines a scalable path to broader mobile deployment of compositional tasks.

Abstract

Large language models (LLMs) are commonly adapted for diverse downstream tasks via parameter-efficient fine-tuning techniques such as Low-Rank Adapters (LoRA). While adapters can be combined to handle multiple tasks separately, standard approaches struggle when targeting the simultaneous execution of complex tasks, such as generating a translated summary from a long conversation. To address this challenge, we propose a novel approach tailored specifically for compositional multi-tasking scenarios involving summarization and translation. Our technique involves adding a learnable projection layer on top of the combined summarization and translation adapters. This design enables effective integration while maintaining efficiency through reduced computational overhead compared to alternative strategies requiring extensive retraining or sequential processing. We demonstrate the practical viability of our method within an on-device environment by developing an Android app capable of executing compositional tasks seamlessly. Experimental results indicate our solution performs well and is fast in both cloud-based and on-device implementations, highlighting the potential benefits of adopting our framework in real-world applications demanding high-speed operation alongside resource constraints.

Figures (6)

• Figure 1: Compositional multi-tasking on the combination of summarization and translation. In our fully on-device system, we focus on the scenario where a conversation gliwa2019samsum in one language is summarized in another language.
• Figure 2: Overview of our projection merge method. We first merge the single-task LoRAs and then pass them through projection parameters.
• Figure 3: The user client for the PoC server-side application. The application operates either in a predefined mode such as two-step LoRA usage (left) or in an experimental mode where all methods are utilized and relevant metrics are reported (right). Tapping on the summary translates it back to English, which has been used in the example on the left side.
• Figure 4: High-level architecture of our on-device system for compositional multi-tasking.
• Figure 5: The application runs entirely on a Samsung S23 Ultra Android device. While similar to the server-side proof-of-concept application, additional information is displayed such as memory consumption.