Towards Understanding the Optimization Mechanisms in Deep Learning

TL;DR

This paper reframes deep learning optimization as conditional distribution estimation under the Fenchel-Young loss $d_{\Omega}$, showing that even though fitting objectives are non-convex, the global optimum can be approached by jointly minimizing the gradient norm and the distribution-fitting error $\mathcal{E}_f$. It provides a theoretical link between gradient-based updates and alignment to the target conditional distribution, and introduces an implicit regularization effect via $d_{\Omega}$ that constrains parameter norms. The structural-error analysis connects network architecture, parameter count, and gradient independence to optimization dynamics, highlighting skip connections and over-parameterization as mechanisms to improve convergence. Empirical validation on MNIST demonstrates the predicted relationships between gradient norms, structural error, and fitting error, and illuminates practical design implications for architecture and initialization. Overall, the work offers a unified, distribution-focused lens for understanding non-convex optimization in finite-width DNNs and motivates further exploration under relaxed independence assumptions.

Abstract

In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.

Figures (9)

• Figure 1: Model architectures and configuration parameters.
• Figure 2: Changes of bounds during the training process.
• Figure 3: Local Pearson correlation coefficient curves during the training process.
• Figure 4: Changes in indicators during the training process.
• Figure 5: Changes in indicators during the increase in model depth.

Theorems & Definitions (12)

• Lemma 1: Properties of Fenchel-Young losses
• Lemma 2
• Lemma 3: Gershgorin’s circle theorem
• Proposition 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Definition 1: Gradient Independence Condition