Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models

Zhuxuanzi Wang, Mingqiao Mo, Xi Xiao, Chen Liu, Chenrui Ma, Yunbei Zhang, Xiao Wang, Smita Krishnaswamy, Tianyang Wang

TL;DR

CTR-LoRA targets parameter-efficient fine-tuning by unifying where to allocate low-rank capacity with how updates move in curvature space. By using curvature proxies (e.g., K-FAC or Hutchinson) to score marginal utility of rank-one directions and enforcing a curvature-based trust region, it achieves higher accuracy, stability, and efficiency without adding inference cost. The approach demonstrates consistent gains across diverse backbones and benchmarks, including out-of-distribution data, and attains a favorable Pareto frontier in accuracy versus memory and throughput. This principled framework advances deployable PEFT by linking rank allocation to update geometry and offering a scalable path for robust adaptation of large language models.

Abstract

Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.

CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models

TL;DR

CTR-LoRA targets parameter-efficient fine-tuning by unifying where to allocate low-rank capacity with how updates move in curvature space. By using curvature proxies (e.g., K-FAC or Hutchinson) to score marginal utility of rank-one directions and enforcing a curvature-based trust region, it achieves higher accuracy, stability, and efficiency without adding inference cost. The approach demonstrates consistent gains across diverse backbones and benchmarks, including out-of-distribution data, and attains a favorable Pareto frontier in accuracy versus memory and throughput. This principled framework advances deployable PEFT by linking rank allocation to update geometry and offering a scalable path for robust adaptation of large language models.

Abstract

Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.
Paper Structure (13 sections, 3 equations, 2 figures, 3 tables)

This paper contains 13 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Definition
  4. Curvature Proxies Second-Order Signals
  5. Curvature-Aware Rank Scheduling
  6. Trust-Region PEFT Objective
  7. Complexity and Memory
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Out-of-Distribution Generalization
  12. Ablation and Diagnostic Analysis
  13. Conclusion

Figures (2)

  • Figure 1: Comparison between LoRA and CTR-LoRA. Left: standard LoRA decomposes the low-rank update $\Delta W=AB^{\top}$ with $A \sim \mathcal{N}(0,\sigma^2)$ and $B=0$, merged into frozen pretrained weights $W \in \mathbb{R}^{d \times d}$. Right: CTR-LoRA introduces curvature-aware rank scheduling (guiding the choice of rank $r$) and a trust-region penalty that regularizes the update trajectory, improving stability and robustness during fine-tuning.
  • Figure 2: Comprehensive analyses of CTR-LoRA. (a-b) Budget-performance curves show consistent improvements. (c) CTR-LoRA has smoother training dynamics and converges to higher accuracy. (d) K-FAC and Hutchinson proxies outperform sensitivity-only allocation, confirming robustness to curvature estimators. (e) Performance is stable within a moderate trust-region range. (f) CTR-LoRA occupies the Pareto frontier by reducing variance, lowering memory, and increasing throughput.