Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge

Sowmya Vajrala, Aakash Parmar, Prasanna R, Sravanth Kodavanti, Manjunath Arveti, Srinivas Soumitri Miriyala, Ashok Senapati

Abstract

Generative Artificial Intelligence (GenAI) features such as image editing, object removal, and prompt-guided image transformation are increasingly integrated into mobile applications. However, deploying Large Vision Models (LVMs) for such tasks on resource-constrained devices remains challenging due to their high memory and compute requirements. While Low-Rank Adapters (LoRAs) enable parameter-efficient task adaptation, existing Mobile deployment pipelines typically compile separate model binaries for each LoRA + a copy of the foundation model, resulting in redundant storage and increased runtime overhead. In this work, we present a unified framework for enabling multi-task GenAI inference on edge devices using a single shared model. Our key idea is to treat LoRA weights as runtime inputs rather than embedding them into the compiled model graph, allowing dynamic task switching at runtime without recompilation. Then, to support efficient on-device execution, we introduce QUAD (Quantization with Unified Adaptive Distillation), a quantizationaware training strategy that aligns multiple LoRA adapters under a shared quantization profile. We implement the proposed system with a lightweight runtime stack compatible with mobile NPUs and evaluate it across multiple chipsets. Experimental results demonstrate up to 6x and 4x reduction in memory footprint and latency improvements, respectively, while maintaining high visual quality across multiple GenAI tasks.

Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge

Abstract

Generative Artificial Intelligence (GenAI) features such as image editing, object removal, and prompt-guided image transformation are increasingly integrated into mobile applications. However, deploying Large Vision Models (LVMs) for such tasks on resource-constrained devices remains challenging due to their high memory and compute requirements. While Low-Rank Adapters (LoRAs) enable parameter-efficient task adaptation, existing Mobile deployment pipelines typically compile separate model binaries for each LoRA + a copy of the foundation model, resulting in redundant storage and increased runtime overhead. In this work, we present a unified framework for enabling multi-task GenAI inference on edge devices using a single shared model. Our key idea is to treat LoRA weights as runtime inputs rather than embedding them into the compiled model graph, allowing dynamic task switching at runtime without recompilation. Then, to support efficient on-device execution, we introduce QUAD (Quantization with Unified Adaptive Distillation), a quantizationaware training strategy that aligns multiple LoRA adapters under a shared quantization profile. We implement the proposed system with a lightweight runtime stack compatible with mobile NPUs and evaluate it across multiple chipsets. Experimental results demonstrate up to 6x and 4x reduction in memory footprint and latency improvements, respectively, while maintaining high visual quality across multiple GenAI tasks.

Paper Structure

This paper contains 10 sections, 8 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. LVM and Multi-LoRA Adaptation
  5. LoRA as Input and one-for-all Frozen Graph
  6. QUAD: Unified LoRA-Compatible Deployment
  7. Compilation and Deployment: Graph Optimization
  8. Experiments and Results
  9. Ablation Analysis
  10. Conclusion

Figures (7)

  • Figure 1: Schematic for a simplified version of compilation and deployment of LVM and depicting the problem definition
  • Figure 2: Pictorial representation of proposed approach depicting a) modification of logic to ensure LoRA as input in the architecture, b) offline construction of a single frozen graph that will serve multiple tasks and c) on-device runtime scenario with the proposed approach
  • Figure 3: Schematic for a simplified version of compilation and deployment of LVM and depicting the problem definition
  • Figure 4: Flowsheet of QUAD framework
  • Figure 5: Software stack for facilitating one-for-all LVMs on embedded devices such as smartphones. In this schematic, we show the stack for on-device deployment of two use-cases with a single LVM as described in Section 4
  • ...and 2 more figures