When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

TL;DR

The paper analyzes how pre-training strength affects LoRA fine-tuning in high-dimensional single-index models and shows that stronger pre-training can delay convergence by extending the correlated search phase. Using a one-pass SGD framework and a generalized Hermite expansion, it derives an exit-time scaling t_exit ~ (τ(μ)/2) log d, with drift coefficients A,B capturing the influence of pre-training and task nonlinearity. The study reveals activation-dependent phenomena: linear and IE=1 activations slow escape as μ increases, while certain Hermite (IE>1) cases exhibit singularities in μ where escape can be blocked; label-squaring can mitigate these effects by effectively reducing the IE to 2. The findings highlight a nuanced trade-off between pre-training richness and algorithmic tractability, suggesting practical strategies (e.g., two-stage label transformations) to harness pre-training while preserving fast transfer dynamics, with implications for parameter-efficient fine-tuning in large models.

Abstract

Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not always true: excessive pre-training can computationally slow down fine-tuning optimization. We study this phenomenon for low-rank adaptation (LoRA) fine-tuning on single-index models trained under one-pass SGD. Leveraging a summary statistics description of the fine-tuning dynamics, we precisely characterize how the convergence rate depends on the initial fine-tuning alignment and the degree of non-linearity of the target task. The key take away is that even when the pre-training and down- stream tasks are well aligned, strong pre-training can induce a prolonged search phase and hinder convergence. Our theory thus provides a unified picture of how pre-training strength and task difficulty jointly shape the dynamics and limitations of LoRA fine-tuning in a nontrivial tractable model.

Figures (12)

• Figure 1: Learning dynamics of the student model \ref{['equ:student_model']} for different levels of pre-training alignment $\mu \in \{0.1, 0.5, 0.8, 0.9\}$, trained on data generated by the teacher model \ref{['equ:teacher_model']}. We consider the matched teacher--student linear activation setting and train the student using one-pass SGD with batch size $B=500$ and input dimension $d=1000$. The panels report the test mean squared error (MSE) (left), the alignment between the student and teacher directions $m=\boldsymbol{\omega}_\star\!\cdot\!\boldsymbol{\omega}$ (middle), and the effective teacher--student overlap $m_{\mathrm{eff}}=\mu+u m$ (right). Shaded regions indicate one standard deviation over three independent runs. The spherical constraint is enforced by normalizing $\boldsymbol{\omega}$ after each gradient step. Additional results for Erf, ReLU, and Sigmoid activations are reported in Appendix \ref{['appendix:numerical evidence of slow down']}.
• Figure 2: Theoretical predictions of the characteristic escape time $\tau(\mu)$ (solid lines) and corresponding numerical estimates (markers) for the time required to exit the correlated search phase of the student model \ref{['equ:student_model']} trained on data generated by the teacher model \ref{['equ:teacher_model']}. Results are shown for matched teacher--student activation functions that have IE $=1$. Numerical results have being obtained by running one-pass SGD with batch-size $B=500,$ input dimension $d=1000$, learning rate $=0.2$ and we average over three instances of the experiment. We record the numerical escape time $\tau_{\rm exp}$ at the point where the overlap reaches $m\simeq 0.98$ for all functions, since the descent phase proceeds exponentially fast (cf. Fig. \ref{['fig:dynamics_activation_comparison_appendix']}).
• Figure 3: Theoretical predictions of the characteristic escape time $\tau(\mu)$ (solid lines) for exiting the correlated search phase of the student model \ref{['equ:student_model']} trained on data generated by the teacher model \ref{['equ:teacher_model']}. Results correspond to matched teacher--student Hermite activations ${\rm He}_k$ (see legend). In the bottom panel, the vertical lines and corresponding dots indicate the specific values of $\mu$ at which numerical simulations are performed and reported in the inset. The inset shows the numerical evolution of the overlap $m$ for ${\rm He}_3$, obtained using one-pass SGD with batch size $B=500$, input dimension $d=1000$, and learning rate $5.5\times 10^{-5}$, at $\mu\in\{0.2,\,0.325\}$ and averaged over three independent runs. These results confirm that, near the singular region $\mu=0.325$ (highlighted by the orange dot), the dynamics require substantially longer times to escape.
• Figure 4: Theoretical predictions (solid lines) of the characteristic escape time $\tau(\mu)$ in a setting where labels generated by the teacher model \ref{['equ:teacher_model']} are squared before training the student model \ref{['equ:student_model']}, thereby modifying the information exponent (IE) of the loss. Top: for standard activations, this transformation reverses the dependence of $\tau(\mu)$ on the signal strength $\mu$ in some cases, while leaving others unchanged. These numerical results are obtained using one-pass SGD with batch size $B=500$, input dimension $d=1000$, and learning rate $lr=0.2$, and are averaged over three independent runs. Bottom: for Hermite activations, squaring enforces $\mathrm{IE}\leq 2$, removes the singular behavior of $\tau(\mu)$, and induces an approximately monotonic decrease of $\tau(\mu)$ with increasing $\mu$; vertical lines and corresponding markers indicate the specific values of $\mu$ at which numerical simulations are performed. Consistently, the inset shows the numerical evolution of the overlap $m$ at the values of $\mu$ highlighted in the main panel, illustrating substantially earlier compare to the unsquared case (cf. Fig. \ref{['fig:escaping_time_pure_Hermite_matching_activation']}), despite identical numerical settings.
• Figure 5: Learning dynamics of the student model \ref{['equ_different_weight']} for different levels of pre-training alignment $\mu \in \{0.1, 0.4, 0.6, 0.9\}$, trained on data generated by the teacher model \ref{['equ:teacher']}. We consider the matched teacher--student Linear, Erf, and ReLU (from top to bottom) activation setting and train the student using one-pass SGD with batch size $B=500$ and input dimension $d=1000$. The panels report the test mean squared error (MSE) (left), the alignment between the student and teacher directions $m=\boldsymbol{\omega}_\star\!\cdot\!\boldsymbol{\omega}$ (middle), and the effective teacher--student overlap $m_{\mathrm{eff}}=\mu+u m$ (right). Shaded regions indicate one standard deviation over three independent runs. The spherical constraint is enforced by normalizing $\boldsymbol{\omega}$ after each gradient step and we used a learning rate $lr=0.2 .$

Theorems & Definitions (13)

• Proposition 1: Exit time from the correlated search phase
• Remark 1
• Proposition 2
• Proposition 3
• proof
• proof
• proof
• Proposition 4
• Remark 4
• Proposition 5