TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction

TL;DR

This work introduces TRSVR, a stochastic trust-region method that integrates SVRG-based variance reduction into a trust-region framework for unconstrained nonconvex finite-sum optimization. It delivers convergence to a first-order stationary point in expectation and achieves iteration and sample complexities matching SVRG-based first-order methods, even when the Hessian geometry is stochastic and possibly data-dependent. The algorithm computes a full gradient only periodically and uses a variance-reduced gradient within a radius proportional to the gradient norm, while allowing Hessian approximations to be stochastic. Empirical results on convex ill-conditioned problems and nonconvex tasks show that TRSVR with curvature information accelerates convergence and outperforms SGD and Adam, with performance sensitive to batch size and inner-loop length. These findings establish TRSVR as a practical, theoretically grounded approach for large-scale stochastic optimization with second-order information.

Abstract

We propose a stochastic trust-region method for unconstrained nonconvex optimization that incorporates stochastic variance-reduced gradients (SVRG) to accelerate convergence. Unlike classical trust-region methods, the proposed algorithm relies solely on stochastic gradient information and does not require function value evaluations. The trust-region radius is adaptively adjusted based on a radius-control parameter and the stochastic gradient estimate. Under mild assumptions, we establish that the algorithm converges in expectation to a first-order stationary point. Moreover, the method achieves iteration and sample complexity bounds that match those of SVRG-based first-order methods, while allowing stochastic and potentially gradient-dependent second-order information. Extensive numerical experiments demonstrate that incorporating SVRG accelerates convergence, and that the use of trust-region methods and Hessian information further improves performance. We also highlight the impact of batch size and inner-loop length on efficiency, and show that the proposed method outperforms SGD and Adam on several machine learning tasks.

Figures (41)

• Figure 1: Optimality Gap with effective passes. Each trajectory represents a method.
• Figure 2: Squared Gradient Norm with effective passes. Each trajectory represents a method.
• Figure 3: Squared gradient norm with epochs on the RCV1 dataset. Each trajectory represents a method.
• Figure 4: Squared gradient norm with epochs on the Mushroom dataset. Each trajectory represents a method.
• Figure 5: Optimality gap with wall-clock time on the Covertype dataset. Each trajectory represents a method.

Theorems & Definitions (20)

• Remark 3.1
• Lemma 3.2: Cauchy Reduction
• Remark 4.3
• Definition 4.4
• Theorem 4.5: Global Convergence
• Corollary 4.6
• Lemma 4.7: One-Step Expected Reduction
• Lemma 4.8: Upper Bound of Variance
• Lemma 1.1
• proof