Mollifying Networks

TL;DR

Deep neural networks struggle with non-convex loss landscapes. The authors introduce mollified objectives that start training on simpler, smoother objectives and progressively reintroduce complexity by annealing a per-layer noise-driven parameter, effectively creating a continuation method with stochastic depth. Empirical results across deep MLPs, CIFAR-10, and LSTM language modeling show faster convergence and improved generalization relative to baselines like ResNet and vanilla LSTMs. This approach connects continuation theory with noisy activations and suggests a practical, regularizing path toward training very deep architectures.

Abstract

The optimization of deep neural networks can be more challenging than traditional convex optimization problems due to the highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks by starting with a smoothed -- or \textit{mollified} -- objective function that gradually has a more non-convex energy landscape during the training. Our proposition is inspired by the recent studies in continuation methods: similar to curriculum methods, we begin learning an easier (possibly convex) objective function and let it evolve during the training, until it eventually goes back to being the original, difficult to optimize, objective function. The complexity of the mollified networks is controlled by a single hyperparameter which is annealed during the training. We show improvements on various difficult optimization tasks and establish a relationship with recent works on continuation methods for neural networks and mollifiers.

Figures (10)

• Figure 1: A sequence of optimization problems of increasing complexity, where the first ones are easy to solve but only the last one corresponds to the actual problem of interest. It is possible to tackle the problems in order, starting each time at the solution of the previous one and tracking the local minima along the way.
• Figure 2: Top: Stochastic depth. Bottom: mollifying network. The dashed line represents the optional residual connection. In the top path, the input is processed with a convolutional block followed by a noisy activation function, while in the bottom path the original activation of the layer $l-1$ is propagated untouched. For each unit, one of the two paths in picked according to a binary stochastic decision $\boldsymbol\pi$.
• Figure 4: The learning curves of a $6$-layers MLP with $\text{sigmoid}$ activation function on $40$ bit parity task.
• Figure 5: The training curve of a bidirectional-RNN that predicts the embedding corresponding to a sequence of characters.
• Figure : a)

Theorems & Definitions (4)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Definition 2.4