Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Neural Network Training with Frank-Wolfe

Sebastian Pokutta, Christoph Spiegel, Max Zimmer

TL;DR

This work investigates projection-free optimization for deep neural networks by applying stochastic Frank–Wolfe methods to training with convex, constrained parameter regions. By systematically evaluating multiple feasible regions (e.g., $L^2$-norm balls, hypercubes, $L^1$-norm-based polytopes, simplices, and permutahedra) and FW variants, it demonstrates that appropriately chosen constraints can yield state-of-the-art or competitive accuracy on CIFAR-10/100 and ImageNet while shaping learned representations and sparsity. The study finds that SFW, especially with momentum, often matches or surpasses unconstrained SGD with weight decay in several benchmarks, and that constraint geometry strongly impacts feature encoding and sparsity patterns. Additionally, the paper provides practical guidance on hyperparameter decoupling from region diameter and discusses the computational trade-offs of various stochastic FW algorithms, making constraint-based training a viable alternative to projection-based methods in deep learning, with code released for reproducibility.

Abstract

This paper studies the empirical efficacy and benefits of using projection-free first-order methods in the form of Conditional Gradients, a.k.a. Frank-Wolfe methods, for training Neural Networks with constrained parameters. We draw comparisons both to current state-of-the-art stochastic Gradient Descent methods as well as across different variants of stochastic Conditional Gradients. In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants. We then show that, by choosing an appropriate region, one can achieve performance exceeding that of unconstrained stochastic Gradient Descent and matching state-of-the-art results relying on $L^2$-regularization. Lastly, we also demonstrate that, besides impacting performance, the particular choice of constraints can have a drastic impact on the learned representations.

Deep Neural Network Training with Frank-Wolfe

TL;DR

This work investigates projection-free optimization for deep neural networks by applying stochastic Frank–Wolfe methods to training with convex, constrained parameter regions. By systematically evaluating multiple feasible regions (e.g., -norm balls, hypercubes, -norm-based polytopes, simplices, and permutahedra) and FW variants, it demonstrates that appropriately chosen constraints can yield state-of-the-art or competitive accuracy on CIFAR-10/100 and ImageNet while shaping learned representations and sparsity. The study finds that SFW, especially with momentum, often matches or surpasses unconstrained SGD with weight decay in several benchmarks, and that constraint geometry strongly impacts feature encoding and sparsity patterns. Additionally, the paper provides practical guidance on hyperparameter decoupling from region diameter and discusses the computational trade-offs of various stochastic FW algorithms, making constraint-based training a viable alternative to projection-based methods in deep learning, with code released for reproducibility.

Abstract

This paper studies the empirical efficacy and benefits of using projection-free first-order methods in the form of Conditional Gradients, a.k.a. Frank-Wolfe methods, for training Neural Networks with constrained parameters. We draw comparisons both to current state-of-the-art stochastic Gradient Descent methods as well as across different variants of stochastic Conditional Gradients. In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants. We then show that, by choosing an appropriate region, one can achieve performance exceeding that of unconstrained stochastic Gradient Descent and matching state-of-the-art results relying on -regularization. Lastly, we also demonstrate that, besides impacting performance, the particular choice of constraints can have a drastic impact on the learned representations.

Paper Structure

This paper contains 28 sections, 3 theorems, 35 equations, 9 figures, 6 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Contributions.
  4. Outline.
  5. Preliminaries
  6. Stochastic Frank--Wolfe algorithms
  7. Regularization via feasible regions
  8. $L^p$-norm ball.
  9. $K$-sparse polytope.
  10. $K$-norm ball.
  11. Unit simplex.
  12. Probability simplex.
  13. Permutahedron.
  14. Technical Considerations
  15. Frank--Wolfe algorithms
  16. ...and 13 more sections

Key Result

Theorem 2.1

Consider the setting of Problem eq:finit-sum-problem and assume that the $\ell_i$ are smooth. If $\rho_t = 1$, $\alpha_t = T^{-1/2}$ and $b_t = T$ for all $0 \leq t < T$ and if $\theta_a$ is chosen uniformly at random from $\{\theta_i : 0 \leq i < T \}$ as determined by Algorithm alg:sfw, then we ha where $\mathbb E$ denotes the expectation w.r.t. all the randomness present.

Figures (9)

  • Figure 1: Comparing different stochastic Frank--Wolfe algorithms to train a fully connected Neural Network with two hidden layers on the Fashion-MNIST dataset.
  • Figure 2: Comparing different stochastic Frank--Wolfe algorithms to train a fully connected Neural Network with one hidden layers on sparse feature vectors generated from the IMDB dataset of movie reviews.
  • Figure 3: Visualization of the weights in a fully connected no-hidden-layer classifier trained on the MNIST dataset corresponding to the digits $0$ and $3$. Red corresponds to negative and green to positive weights. The unconstrained network in the first column was trained using SGD and the constrained networks in the remaining columns were trained using SFW.
  • Figure 4: Number of active parameters vs. test set error in two different networks trained on the MNIST dataset. When the parameters are contrained to lie in a specified feasible region, the networks were trained using SFW with momentum. In the unconstrained case they were trained using SGD. Results are averaged over 5 runs. The full setup is in Appendix \ref{['app:computations']}.
  • Figure 5: Comparing different stochastic Frank--Wolfe algorithms to train a fully connected Neural Network on the MNIST dataset. The fully connected Neural Network consists of two hidden layers of size 64 with ReLU activations. The $L^2$-constraints were determined according to Equation \ref{['eq:initialization_diameter']} with $w = 300$. For SFW batch size was 50 and learning rate was $0.3$ set according to Equation \ref{['eq:decouple_diameter']}. For MSFW batch size was 50, learning rate was $0.1$ set according to Equation \ref{['eq:decouple_diameter']}, and momentum was $0.9$, that is $\rho = 0.1$. For SVRF batch size was 50 and learning rate was $0.1$ set according to Equation \ref{['eq:decouple_diameter']}, and the reference point was updated at the beginning of each epoch. For ORGFW batch size was 50, learning rate was $0.03$ set according to Equation \ref{['eq:decouple_diameter']}, and momentum was $0.9$, that is $\rho = 0.1$. Results are averaged over 10 runs each.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 2.1: ReddiSraPoczosBarnabasSmola2016
  • Theorem A.1: ReddiSraPoczosBarnabasSmola2016
  • Lemma A.2
  • proof : Proof of Theorem \ref{['thm:sfw_convergence_extended']}