Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A multiobjective continuation method to compute the regularization path of deep neural networks

Augustina C. Amakor, Konstantin Sonntag, Sebastian Peitz

TL;DR

This work tackles the problem of computing the regularization path for deep neural networks by treating training loss and sparsity as a nonconvex multiobjective optimization problem. It introduces a predictor-corrector continuation method based on a Multiobjective Proximal Gradient (MPG) framework to efficiently trace the Pareto front in very high-dimensional settings, despite the non-smooth $\ell^1$ term. The authors demonstrate the method on Iris, MNIST, and CIFAR10, showing that starting from sparse networks yields well-distributed fronts and improved generalization, with substantial computational advantages over weighted-sum and NSGA-II approaches. Overall, the paper extends regularization-path concepts to nonlinear, large-scale DNNs, offering a scalable path toward parsimonious models and task-adaptive architectures with practical impact for efficient ML.

Abstract

Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a \emph{regularization path} connecting the sparsest solution in terms of the $\ell^1$ norm, i.e., zero weights and the non-regularized solution. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity ($\ell^1$ norm) as two conflicting criteria and solving the resulting multiobjective optimization problem for low-dimensional DNN. However, due to the non-smoothness of the $\ell^1$ norm and the high number of parameters, this approach is not very efficient from a computational perspective for high-dimensional DNNs. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. To the best of our knowledge, this is the first algorithm to compute the regularization path for non-convex multiobjective optimization problems (MOPs) with millions of degrees of freedom.

A multiobjective continuation method to compute the regularization path of deep neural networks

TL;DR

This work tackles the problem of computing the regularization path for deep neural networks by treating training loss and sparsity as a nonconvex multiobjective optimization problem. It introduces a predictor-corrector continuation method based on a Multiobjective Proximal Gradient (MPG) framework to efficiently trace the Pareto front in very high-dimensional settings, despite the non-smooth term. The authors demonstrate the method on Iris, MNIST, and CIFAR10, showing that starting from sparse networks yields well-distributed fronts and improved generalization, with substantial computational advantages over weighted-sum and NSGA-II approaches. Overall, the paper extends regularization-path concepts to nonlinear, large-scale DNNs, offering a scalable path toward parsimonious models and task-adaptive architectures with practical impact for efficient ML.

Abstract

Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a \emph{regularization path} connecting the sparsest solution in terms of the norm, i.e., zero weights and the non-regularized solution. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity ( norm) as two conflicting criteria and solving the resulting multiobjective optimization problem for low-dimensional DNN. However, due to the non-smoothness of the norm and the high number of parameters, this approach is not very efficient from a computational perspective for high-dimensional DNNs. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. To the best of our knowledge, this is the first algorithm to compute the regularization path for non-convex multiobjective optimization problems (MOPs) with millions of degrees of freedom.
Paper Structure (18 sections, 1 theorem, 7 equations, 8 figures, 1 table, 2 algorithms)

This paper contains 18 sections, 1 theorem, 7 equations, 8 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Multiobjective optimization
  4. Multicriteria machine learning
  5. Continuation Method
  6. Some basics on multiobjective optimization
  7. Multiobjective proximal gradient method
  8. A predictor-corrector method
  9. Predictor step
  10. Corrector step
  11. Numerical experiments
  12. Experimental settings
  13. Experimental Procedure
  14. Results
  15. MNIST
  16. ...and 3 more sections

Key Result

Theorem 6

Tanabe2019 Let $f_i$ be convex with $L$-Lipschitz continuous gradients and let $g_i$ be proper, convex and lower semi-continuous for all $i=1, \dots ,m$ with step size $h \leq \frac{2}{L}$. Then, every accumulation point of the sequence $\{\theta^k \}$ computed by Algorithm algo:example1 is Pareto c

Figures (8)

  • Figure 1: Contour plot of smooth objective functions $F_1(\theta) = \frac{1}{2}\|A\theta - b \|^2_2$ and non-smooth objective function $F_2(\theta) = \lVert \theta \rVert_1$ in black and the regularization path$\theta^*(\lambda) \in \text{argmin}_{\theta \in \mathbb{R}^n} \frac{1}{2}\lVert A \theta - b \rVert_2^2 + \lambda\lVert \theta \rVert_1$ in red.
  • Figure 2: Sketch of the continuation method; The predictor steps are shown in black and blue for Eqs. \ref{['eq:gradient']} and \ref{['eq:shrinkage']} respectively. The corrector step is shown in the red lines and the red dots are the Pareto optimal points computed by the predictor-corrector method.
  • Figure 3: The neural network architecture used for testing the CM and WS methods on CIFAR10 dataset.
  • Figure 4: Pareto front approximation for the Iris dataset using Algorithm 2 (red symbols) versus the reference Pareto front in "blue" (computed using the same algorithm with very small step sizes and many different initial conditions) with unscaled $\ell^1$ norm.
  • Figure 5: (\ref{['fig:sub3']}) The Pareto front for the MNIST dataset (in black), where the initial point is shown in blue. The red curve shows the performance on the validation set. Non-sparse networks clearly tend to overfit. (\ref{['fig:sub4']}) The prediction accuracy versus $\ell^1$ norm, where the overfitting regime becomes apparent once more.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Definition 1
  • Definition 2
  • Remark 3
  • Definition 4: Proximal operator
  • Remark 5
  • Theorem 6