Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies

TL;DR

This work addresses efficient fine-tuning of large language models under memory constraints by formalizing the Fidelity-Plasticity Trade-off: a layer's learning potential is bounded by the information capacity of its frozen weights. It introduces QR-Adaptor, a gradient-free framework that jointly optimizes per-layer quantization bit-widths and LoRA ranks, guided by a three-stage pipeline: task-informed Fidelity Sensitivity Profiling, discrete global landscape exploration, and Bayesian frontier refinement. Empirical results across Qwen and LLaMA families show that mixed-precision configurations discovered by QR-Adaptor establish a new Pareto frontier, with a 4-bit budget rivaling 16-bit baselines and strong performance on both general NLU and GSM8K reasoning tasks. The method also reveals a consistent linguistic hierarchy in resource allocation, allocating higher fidelity and plasticity to deeper semantic layers while compressing syntactic layers, thus enabling effective edge deployment and informing future design of quantized-tuning strategies.

Abstract

Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. While Quantization-Aware Fine-tuning has emerged as a viable solution, existing paradigms typically decouple quantization and adapter optimization. This separation overlooks a fundamental theoretical constraint we identify as the \textit{Fidelity-Plasticity Trade-off}: a layer's capacity to adapt to new tasks (Plasticity) is inherently constrained by the information capacity of its frozen weights (Fidelity). Aggressively quantizing semantically critical layers creates an information bottleneck that no amount of adapter rank can recover, while high precision in robust syntactic layers wastes valuable memory. To address this, we introduce \textbf{QR-Adaptor}, a unified framework that jointly optimizes per-layer quantization bit-width and LoRA rank. By formulating resource allocation as a multi-objective search aligned with the model's linguistic hierarchy, our method systematically liberates memory from redundancy-heavy layers to reinvest in capacity-critical ones. Extensive experiments demonstrate that QR-Adaptor establishes a new Pareto frontier: notably, a model fine-tuned under a strict 4-bit memory budget achieves performance rivaling 16-bit baselines, demonstrating that precise resource alignment is as critical as model size.

Figures (6)

• Figure 1: Empirical validation of the Fidelity-Plasticity Trade-off. We compare four configurations on Qwen3-1.7B. Crucially, despite having the same memory budget, Config D (High Fidelity in Deep Layers) significantly outperforms Config C (High Fidelity in Shallow Layers). This confirms our hypothesis that deep semantic layers are physically gated by quantization noise, requiring strategic bit allocationa.
• Figure 2: Overview of the QR-Adaptor Framework. The pipeline consists of three synergistic stages: (I) Fidelity Sensitivity Profiling initializes the population based on information entropy to respect layer-wise task demand; (II) Discrete Landscape Exploration utilizes a constrained evolutionary strategy to approximate the global Pareto frontier without gradient mismatch; (III) Bayesian Frontier Refinement employs Gaussian Process regression to pinpoint the optimal bit-rank configuration within the non-smooth solution space.
• Figure 3: Layer-wise Bit-Rank Allocation. The discovered configuration exhibits a clear gradient: high fidelity (bits) and plasticity (rank) are automatically concentrated in deep semantic layers, while shallow layers are aggressively compressed.
• Figure 4: Search Convergence. Validation PPL decreases steadily, proving the effectiveness of the evolutionary exploration and Bayesian refinement.
• Figure 5: Impact of Joint Optimization. Joint optimization yields the better trade-off.