On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

TL;DR

This work tackles the challenge of deploying quantized large language models on edge devices by enabling on-the-fly adaptation of LoRA adapters to arbitrary per-layer quantization configurations. It introduces CoA-LoRA, a configuration-aware model that maps quantization settings to lightweight per-layer adjustments, paired with a Pareto-based Gaussian-process search to curate a diverse, high-quality training configuration set. A cyclic training procedure jointly optimizes the configuration-aware mapper and expands the training configurations via gradient-guided search and diversity-preserving filtering, eliminating the need for separate fine-tuning for each configuration. Empirically, CoA-LoRA delivers competitive or superior accuracy with significantly reduced fine-tuning time across multiple models and tasks, while demonstrating strong generalization to unseen configurations. The approach promises practical impact for deploying quantized LLMs across heterogeneous devices, enabling efficient, scalable, and robust fine-tuning without repetitive per-config tuning.

Abstract

As increasingly large pre-trained models are released, deploying them on edge devices for privacy-preserving applications requires effective compression. Recent works combine quantization with the fine-tuning of high-precision LoRA adapters, which can substantially reduce model size while mitigating the accuracy loss from quantization. However, edge devices have inherently heterogeneous capabilities, while performing configuration-wise fine-tuning for every quantization setting is computationally prohibitive. In this paper, we propose CoA-LoRA, a method that dynamically adjusts the LoRA adapter to arbitrary quantization configurations (i.e., the per-layer bit-width choices of a pre-trained model) without requiring repeated fine-tuning. This is accomplished via a configuration-aware model that maps each configuration to its low-rank adjustments. The effectiveness of this model critically depends on the training configuration set, a collection of configurations chosen to cover different total bit-width budgets. However, constructing a high-quality configuration set is non-trivial. We therefore design a Pareto-based configuration search that iteratively optimizes the training configuration set, yielding more precise low-rank adjustments. Our experiments demonstrate that, unlike the state-of-the-art methods that require fine-tuning a separate LoRA adapter for each configuration, CoA-LoRA incurs no additional time cost while achieving comparable or even superior performance to those methods.

Figures (12)

• Figure 1: Accuracy gap (left) and performance comparison of cumulative fine-tuning time (right) on the SST-2 task from the GLUE benchmark using RoBERTa-Large model.
• Figure 2: CoA-LoRA workflow: optimizing both the quantization configurations and the configuration-aware model to achieve maximum accuracy at any given bit-width.
• Figure 3: Illustration of configuration-aware LoRA adapters with parallel adjustment. The configuration-aware model $\boldsymbol{\theta}$ generates adjustment matrices $\mathbf{I} + \mathbf{U}_{\boldsymbol{\theta}}(\mathbf{C}_i)$ from the quantization configuration $\mathbf{C}_i$ in parallel, where $\mathbf{I}$ denotes the identity matrix.
• Figure 4: Illustration of the Hypervolume Improvement (left) and the Segmented Pareto Front (right). In the right figure, red points indicate the Pareto-optimal configurations, blue points are the configurations preserved in the final set after segmentation, and yellow points are discarded suboptimal configurations.
• Figure 5: Comparison of accuracy across four tasks under different bit-widths.