Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parle: parallelizing stochastic gradient descent

Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, Ameet Talwalkar, Adam Oberman

TL;DR

Parle presents a parallel training algorithm that interleaves local-entropy optimization with elastic coupling to a central reference, enabling infrequent communication and strong generalization via flat minima. It unifies Entropy-SGD and Elastic-SGD to train multiple replicas that converge to a high-overlap region, producing a single robust model with substantial speedups over data-parallel SGD. Empirically, Parle achieves near state-of-the-art error rates on CIFAR-10/100 and competitive results on MNIST and SVHN, while maintaining modest communication overhead and robustness to hyper-parameters. This approach offers a practical path to scalable, distributed non-convex optimization for deep networks, with potential extensions to heterogeneous hardware and private-data settings.

Abstract

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

Parle: parallelizing stochastic gradient descent

TL;DR

Parle presents a parallel training algorithm that interleaves local-entropy optimization with elastic coupling to a central reference, enabling infrequent communication and strong generalization via flat minima. It unifies Entropy-SGD and Elastic-SGD to train multiple replicas that converge to a high-overlap region, producing a single robust model with substantial speedups over data-parallel SGD. Empirically, Parle achieves near state-of-the-art error rates on CIFAR-10/100 and competitive results on MNIST and SVHN, while maintaining modest communication overhead and robustness to hyper-parameters. This approach offers a practical path to scalable, distributed non-convex optimization for deep networks, with potential extensions to heterogeneous hardware and private-data settings.

Abstract

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

Paper Structure

This paper contains 20 sections, 17 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Motivation
  4. Background and related work
  5. Entropy-SGD
  6. Elastic-SGD
  7. Equivalence of Entropy-SGD and Elastic-SGD
  8. Scoping
  9. Related work
  10. Parle
  11. Hyper-parameter choice
  12. Many deputies under one sheriff
  13. Empirical validation
  14. Implementation details and communication latencies
  15. MNIST
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Permutation invariant overlap of independently trained networks
  • Figure 2: Validation error: $\mathrm{LeNet}$ on MNIST
  • Figure 3: Validation error of $\textrm{WRN-28-10}$ on CIFAR-10 (\ref{['fig:wrn_cifar10_full_valid']}) and CIFAR-100 (\ref{['fig:wrn_cifar100_full_valid']})
  • Figure 4: Validation error: $\textrm{WRN-16-4}$ on the SVHN dataset
  • Figure 5: Training error on CIFAR-10 (\ref{['fig:wrn_cifar10_full_valid']}), CIFAR-100 (\ref{['fig:wrn_cifar100_full_valid']}) and SVHN (\ref{['fig:wrn_svhn_full_train']})
  • ...and 1 more figures

Theorems & Definitions (4)

  • Remark 1: Scoping and learning rate annealing
  • Remark 2: Nesterov's momentum
  • Remark 3: Running Parle on diverse computational platforms
  • Remark 4: Plotting against wall-clock time