Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoRA Dropout as a Sparsity Regularizer for Overfitting Control

Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, Hong Mei

TL;DR

This paper addresses overfitting in LoRA-based PEFT for large PLMs. It introduces LoRA Dropout, which applies stochastic masks to low-rank factors, inducing sparsity and enabling a sparsity-regularization view with a generalization bound that depends on $(2p-p^2)$. It further introduces a test-time ensemble to tighten the error bound and improve test-time generalization. Empirical results on GLUE, SQuAD, and instruction-tuning show improved accuracy and calibration, with a noted trade-off in computational overhead.

Abstract

Parameter-efficient fine-tuning methods, represented by LoRA, play an essential role in adapting large-scale pre-trained models to downstream tasks. However, fine-tuning LoRA-series models also faces the risk of overfitting on the training dataset, and yet there's still a lack of theoretical guidance and practical mechanism to control overfitting on LoRA-based PEFT methods. In this paper, we propose a LoRA Dropout mechanism for the LoRA-based methods by introducing random noises to the learnable low-rank matrices and increasing parameter sparsity. We then demonstrate the theoretical mechanism of our LoRA Dropout mechanism from the perspective of sparsity regularization by providing a generalization error bound under this framework. Theoretical results show that appropriate sparsity would help tighten the gap between empirical and generalization risks and thereby control overfitting. Furthermore, based on the LoRA Dropout framework, we introduce a test-time ensemble strategy and provide theoretical evidence demonstrating that the ensemble method can further compress the error bound, and lead to better performance during inference time. Extensive experiments on various NLP tasks provide practical validations of the effectiveness of our LoRA Dropout framework in improving model accuracy and calibration.

LoRA Dropout as a Sparsity Regularizer for Overfitting Control

TL;DR

This paper addresses overfitting in LoRA-based PEFT for large PLMs. It introduces LoRA Dropout, which applies stochastic masks to low-rank factors, inducing sparsity and enabling a sparsity-regularization view with a generalization bound that depends on . It further introduces a test-time ensemble to tighten the error bound and improve test-time generalization. Empirical results on GLUE, SQuAD, and instruction-tuning show improved accuracy and calibration, with a noted trade-off in computational overhead.

Abstract

Parameter-efficient fine-tuning methods, represented by LoRA, play an essential role in adapting large-scale pre-trained models to downstream tasks. However, fine-tuning LoRA-series models also faces the risk of overfitting on the training dataset, and yet there's still a lack of theoretical guidance and practical mechanism to control overfitting on LoRA-based PEFT methods. In this paper, we propose a LoRA Dropout mechanism for the LoRA-based methods by introducing random noises to the learnable low-rank matrices and increasing parameter sparsity. We then demonstrate the theoretical mechanism of our LoRA Dropout mechanism from the perspective of sparsity regularization by providing a generalization error bound under this framework. Theoretical results show that appropriate sparsity would help tighten the gap between empirical and generalization risks and thereby control overfitting. Furthermore, based on the LoRA Dropout framework, we introduce a test-time ensemble strategy and provide theoretical evidence demonstrating that the ensemble method can further compress the error bound, and lead to better performance during inference time. Extensive experiments on various NLP tasks provide practical validations of the effectiveness of our LoRA Dropout framework in improving model accuracy and calibration.
Paper Structure (31 sections, 7 theorems, 32 equations, 5 figures, 9 tables, 1 algorithm)

This paper contains 31 sections, 7 theorems, 32 equations, 5 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Parameter-Efficient Fine-Tuning (PEFT)
  4. Dropout Regularization
  5. Proposed LoRA Dropout Framework
  6. Background: Low Rank Adaptation (LoRA)
  7. LoRA Dropout
  8. Training Objective
  9. Test-time Ensemble
  10. Theoretical Results
  11. LoRA Dropout Fine-Tuning Through the Lens of Sparse Regularization
  12. Generalization Analysis
  13. Error Bound of LoRA Dropout Ensemble Classifier
  14. Experiments
  15. Natural Language Understanding
  16. ...and 16 more sections

Key Result

Proposition 4.2

If the loss function $\mathcal{L}_\lambda$ of LoRA Dropout algorithm $\mathcal{M}$ is $\eta$-Lipschitz, and $\bm{\theta}_{\mathcal{L}_\lambda} (\mathbf{S}^i)$ is close to $\bm{\theta}_{\mathcal{L}_\lambda} (\mathbf{S})$, the Hessian matrix $\nabla^2 \mathcal{L}(\bm{\theta}_{\mathcal{L}_\lambda} (\ma

Figures (5)

  • Figure 1: Loss curves on train and test set of SST2 dataset during fine-tuning of (a) LoRA w/wo our dropout framework, (b) AdaLoRA w/wo our dropout framework.
  • Figure 2: Our proposed dropout framework combined with both LoRA and AdaLoRA methods.
  • Figure 3: The Expected Calibration Error (ECE $\downarrow$) during the fine-tuning process of RTE task.
  • Figure 4: Ablation studies on the dropout strategy.
  • Figure 5: Sensitive analysis on dropout rate and sample number.

Theorems & Definitions (11)

  • Definition 4.1: Pointwise Hypothesis Stability bousquet2002stability
  • Proposition 4.2: PHS Upper Bound of LoRA Dropout
  • Lemma 4.3
  • Theorem 4.4: LoRA Dropout Generalization Error Bound
  • Theorem 4.5: Error Bound of Test-time LoRA Dropout Ensemble
  • Lemma 1.1
  • proof
  • Proposition 1.2: PHS Upper Bound of LoRA Dropout
  • proof
  • Theorem 1.3: Error Bound of Bayes LoRA Dropout Ensemble
  • ...and 1 more