Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network

Zih-Syuan Huang, Ching-pei Lee

TL;DR

RAMDA introduces a practical regularized adaptive momentum dual-averaging method for training structured neural networks, coupling a diagonal preconditioner with an inexact subproblem solver. It proves that iterates identify the regulator-induced structure at the stationary point, even under inexact subproblem solutions, and provides an efficient proximal-gradient solver to enforce the inexactness criteria. The approach yields strong empirical performance across computer vision, language modeling, and speech tasks while achieving higher structured sparsity. This framework offers a scalable path to training models with desirable local structure and improved predictive accuracy.

Abstract

We propose a Regularized Adaptive Momentum Dual Averaging (RAMDA) algorithm for training structured neural networks. Similar to existing regularized adaptive methods, the subproblem for computing the update direction of RAMDA involves a nonsmooth regularizer and a diagonal preconditioner, and therefore does not possess a closed-form solution in general. We thus also carefully devise an implementable inexactness condition that retains convergence guarantees similar to the exact versions, and propose a companion efficient solver for the subproblems of both RAMDA and existing methods to make them practically feasible. We leverage the theory of manifold identification in variational analysis to show that, even in the presence of such inexactness, the iterates of RAMDA attain the ideal structure induced by the regularizer at the stationary point of asymptotic convergence. This structure is locally optimal near the point of convergence, so RAMDA is guaranteed to obtain the best structure possible among all methods converging to the same point, making it the first regularized adaptive method outputting models that possess outstanding predictive performance while being (locally) optimally structured. Extensive numerical experiments in large-scale modern computer vision, language modeling, and speech tasks show that the proposed RAMDA is efficient and consistently outperforms state of the art for training structured neural network. Implementation of our algorithm is available at https://www.github.com/ismoptgroup/RAMDA/.

Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network

TL;DR

RAMDA introduces a practical regularized adaptive momentum dual-averaging method for training structured neural networks, coupling a diagonal preconditioner with an inexact subproblem solver. It proves that iterates identify the regulator-induced structure at the stationary point, even under inexact subproblem solutions, and provides an efficient proximal-gradient solver to enforce the inexactness criteria. The approach yields strong empirical performance across computer vision, language modeling, and speech tasks while achieving higher structured sparsity. This framework offers a scalable path to training models with desirable local structure and improved predictive accuracy.

Abstract

We propose a Regularized Adaptive Momentum Dual Averaging (RAMDA) algorithm for training structured neural networks. Similar to existing regularized adaptive methods, the subproblem for computing the update direction of RAMDA involves a nonsmooth regularizer and a diagonal preconditioner, and therefore does not possess a closed-form solution in general. We thus also carefully devise an implementable inexactness condition that retains convergence guarantees similar to the exact versions, and propose a companion efficient solver for the subproblems of both RAMDA and existing methods to make them practically feasible. We leverage the theory of manifold identification in variational analysis to show that, even in the presence of such inexactness, the iterates of RAMDA attain the ideal structure induced by the regularizer at the stationary point of asymptotic convergence. This structure is locally optimal near the point of convergence, so RAMDA is guaranteed to obtain the best structure possible among all methods converging to the same point, making it the first regularized adaptive method outputting models that possess outstanding predictive performance while being (locally) optimally structured. Extensive numerical experiments in large-scale modern computer vision, language modeling, and speech tasks show that the proposed RAMDA is efficient and consistently outperforms state of the art for training structured neural network. Implementation of our algorithm is available at https://www.github.com/ismoptgroup/RAMDA/.
Paper Structure (26 sections, 4 theorems, 64 equations, 2 figures, 9 tables, 3 algorithms)

This paper contains 26 sections, 4 theorems, 64 equations, 2 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dual Averaging for Deep Learning.
  4. Regularized Stochastic Algorithms for Deep Learning.
  5. Structure and Manifold Identification.
  6. Problem Setting and Algorithm
  7. Subproblem Solver
  8. Analysis
  9. Experiments
  10. Subproblem
  11. Image Classification
  12. Language Modeling
  13. Speech Synthesis
  14. Time Efficiency
  15. Summary
  16. ...and 11 more sections

Key Result

Theorem 1

Assume that eq:prox and eq:proxgen has at least one optimal solution with a finite optimal objective value. Given $\epsilon_t > 0$, the number of iterations alg:solver takes to satisfy eq:inexact for both eq:prox and eq:proxgen is $O(\log(\epsilon_t^{-1}))$ when $\psi$ is convex and $O(\epsilon_t^{-

Figures (2)

  • Figure 1: Group sparsity level and validation prediction performance v.s epochs. In the plot for Transformer-XL, one step processes ten batches, and for our batch size of 64, one epoch consists of 8,401 batches.
  • Figure 2: Group sparsity level at the last epochs.

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Definition 1: Partial Smoothness Lew02aHarL04a
  • Definition 2: Prox-regularity RP96a
  • Theorem 3
  • Theorem 4
  • proof
  • proof
  • proof
  • proof