A Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization

TL;DR

The paper addresses the theoretical understanding of optimization for training deep learning models, surveying gradient-based methods, landscape-aware optimizers, and distributed schemes through convex and nonconvex lenses. It presents a unified view of convergence rates, stability-based generalization bounds, and the computational trade-offs of first- and second-order methods, as well as centralized and decentralized distributed approaches. It highlights landscape-aware strategies like SAM, Entropy-SGD, LPF-SGD, and SmoothOut, clarifying their empirical generalization benefits and, where available, theoretical guarantees. Overall, the work aims to serve as a theoretical handbook, linking core optimization theory to practical DL training challenges and guiding future research by clarifying when and why different methods may generalize better or converge faster.

Abstract

As data sets grow in size and complexity, it is becoming more difficult to pull useful features from them using hand-crafted feature extractors. For this reason, deep learning (DL) frameworks are now widely popular. The Holy Grail of DL and one of the most mysterious challenges in all of modern ML is to develop a fundamental understanding of DL optimization and generalization. While numerous optimization techniques have been introduced in the literature to navigate the exploration of the highly non-convex DL optimization landscape, many survey papers reviewing them primarily focus on summarizing these methodologies, often overlooking the critical theoretical analyses of these methods. In this paper, we provide an extensive summary of the theoretical foundations of optimization methods in DL, including presenting various methodologies, their convergence analyses, and generalization abilities. This paper not only includes theoretical analysis of popular generic gradient-based first-order and second-order methods, but it also covers the analysis of the optimization techniques adapting to the properties of the DL loss landscape and explicitly encouraging the discovery of well-generalizing optimal points. Additionally, we extend our discussion to distributed optimization methods that facilitate parallel computations, including both centralized and decentralized approaches. We provide both convex and non-convex analysis for the optimization algorithms considered in this survey paper. Finally, this paper aims to serve as a comprehensive theoretical handbook on optimization methods for DL, offering insights and understanding to both novice and seasoned researchers in the field.

Figures (8)

• Figure 1: The DL optimization methods discussed in this survey paper.
• Figure 2: Figures for SGD with and without momentum. This Figure is taken from ruder2016overview (Figure 2).
• Figure 3: Consider the train and test loss (resp. $F_{\text{train}}$ and $F_{\text{test}}$), which have similar shape but are shifted with respect to each other. The local minimum that lies in the flat region of the landscape ($x^{*}_{\text{flat}}$) admits a similar value of the train and test loss, despite the shift between them, and thus generalizes well (in fact this property holds for network parameters that lie close enough to $x^{*}_{\text{flat}}$ as well). The local minimum that lies in the narrow region of the landscape (denoted as $x^{*}_{\text{non-flat}}$) admits a significantly larger value of the test loss compared to the train loss, due to the shift between them, and thus generalizes very poorly.
• Figure 4: Local entropy concentrates on wide valleys in the energy landscape (also interpreted as loss function). This figure is taken from DBLP:journals/corr/ChaudhariCSL17 (Figure 2).
• Figure 5: Dowpour SGD. Model replicas asynchronously fetch parameters and push gradients to the parameter server. The figure is taken from Figure 2 of dean2012large

Theorems & Definitions (106)

• Definition 1: strongly convex
• Definition 2: L-Lipschitz continuity
• Definition 3: L-smoothness
• Definition 4: Sublinear convergence rate
• Definition 5: Linear convergence rate
• Definition 6: Superlinear convergence rate
• Definition 7: Quadratic convergence rate
• Definition 8: $\epsilon_s$-uniform stability
• Theorem 9: ramezani2018stability
• Theorem 10: Convergence of SGD; Convex Setting