Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning

TL;DR

This work explains why LoRA-based parameter-efficient fine-tuning is robust to label noise. It develops a theoretical framework built on a memorization capacity bound, a rank-robustness bias–variance tradeoff, and a temporal separation of clean pattern learning from noise memorization, all under a rank-bounded update to pretrained weights. Building on this theory, it introduces RACT, a Rank-Aware Curriculum Training algorithm that uses rank discrepancy between low- and high-rank adapters to detect mislabeled samples, and demonstrates competitive classification accuracy while achieving strong noise-detection performance on NLP benchmarks and reasonable results on vision tasks. The findings offer practical guidance for rank selection, early stopping strategies, and dataset curation, enabling more reliable fine-tuning of large models in noisy-data regimes.

Abstract

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability.

Figures (7)

• Figure 1: Memorization capacity scales with rank. Low-rank adapters (green region) cannot memorize noise; high-rank adapters cross the noise threshold.
• Figure 2: Temporal separation in LoRA training. Clean patterns are learned early; noise memorization occurs later. Separation depends on rank $r$ and noise rate $\eta$.
• Figure 3: RACT architecture. Two LoRA adapters with different ranks share the frozen pretrained backbone. Prediction disagreement identifies noisy samples.
• Figure 4: Classification accuracy comparison across datasets at 30% noise. RACT achieves competitive accuracy with baselines: best on CIFAR-10 (47.36%), comparable on AG News (91.46% vs Label Smoothing's 91.66%), and lower on IMDB (86.47% vs CE's 88.01%). Unlike baselines, RACT provides noise detection capability.
• Figure 5: Noise detection F1 scores. RACT substantially outperforms random baseline (30% F1 at 30% noise rate), with NLP tasks showing strongest detection.

Theorems & Definitions (18)

• Remark 3.1: Scope of Theoretical Results
• Definition 3.2: Memorization
• Theorem 3.3: Memorization Capacity Bound
• proof : Proof Sketch
• Remark 3.4: Existential Nature of Capacity Bounds
• Remark 3.5
• Theorem 3.7: Rank-Robustness Tradeoff
• Remark 3.8: Extension to Classification
• Remark 3.9: Notation Convention
• proof : Proof Sketch