Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression

TL;DR

This paper demonstrates that gradient descent with a large, constant stepsize can accelerate optimization for $\ell_2$-regularized logistic regression in the edge-of-stability regime, achieving $\tilde{O}(\sqrt{\kappa})$-like performance in the small-\lambda regime and $\tilde{O}(\lambda^{-2/3})$ in general, all while the objective may oscillate nonmonotonically. The results extend to population risk under separable distributions, yielding near-statistical-bottleneck performance in $\tilde{O}(n^{2/3})$ steps, and establish a critical stepsize threshold $\eta_{\mathrm{crit}} = 1/(\lambda \ln(1/\lambda))$ that separates convergence from divergence in general settings. A 1D global-convergence result and a lower-bound for stable convergence complement the theory, highlighting that acceleration relies on entering the EoS phase. Overall, the work shows that aggressive, fixed stepsizes can substantially reduce computation under regularization and statistical uncertainty, motivating further exploration of large-step GD across broader losses and models.

Abstract

We study gradient descent (GD) with a constant stepsize for $\ell_2$-regularized logistic regression with linearly separable data. Classical theory suggests small stepsizes to ensure monotonic reduction of the optimization objective, achieving exponential convergence in $\widetilde{\mathcal{O}}(κ)$ steps with $κ$ being the condition number. Surprisingly, we show that this can be accelerated to $\widetilde{\mathcal{O}}(\sqrtκ)$ by simply using a large stepsize -- for which the objective evolves nonmonotonically. The acceleration brought by large stepsizes extends to minimizing the population risk for separable distributions, improving on the best-known upper bounds on the number of steps to reach a near-optimum. Finally, we characterize the largest stepsize for the local convergence of GD, which also determines the global convergence in special scenarios. Our results extend the analysis of Wu et al. (2024) from convex settings with minimizers at infinity to strongly convex cases with finite minimizers.

Figures (3)

• Figure 1: The effect of the stepsize ($\eta$) for GD in logistic regression with $\ell_2$-regularization ($\lambda$). Here, "CV" stands for convergence and "DV" stands for divergence.
• Figure 2: Illustration of large stepsizes accelerating GD. We run constant stepsize GD for an $\ell_2$-regularized logistic regression on a two-dimensional separable dataset. The dataset is given by $\mathbf{x}_1 = (\gamma, 1)$, $\mathbf{x}_2 = (\gamma, -2)$, $y_1=y_2=1$, where $\gamma = 0.2$. The regularization is $\lambda = 2^{-12}$. GD is initialized at $\mathbf{w}_0=0$. Left: Objective value as a function of training steps. Middle: Sharpness (the largest eigenvalue of the Hessian of the objective) as a function of training steps. Right: GD trajectory in the parameter space, where the black dot is the GD initialization and the black cross is the minimizer. Additional details and plots are given in \ref{['apx:exp']}.
• Figure 3: Additional plots for the $2$-dimensional experiment. Top left: Objective value as a function of training steps. Top right: Objective value as a function of training steps for even larger stepsizes. Bottom left: Value of the regularization component as a function of training steps. Bottom right: Value of the logistic component as a function of training steps.

Theorems & Definitions (55)

• Theorem 1: Convergence under small regularization
• Corollary 2: Step complexity under small regularization
• Theorem 3: A lower bound
• Theorem 4: Convergence under general regularization
• Corollary 5: Step complexity under general regularization
• Lemma 1: EoS bounds
• Proposition 6: A population risk bound
• Theorem 7: The critical stepsize
• Conjecture 1
• Lemma 2: Self-boundedness of the logistic function