Failures of Gradient-Based Deep Learning

TL;DR

The paper analyzes systematic failures of gradient-based deep learning on deliberately simple problems, showing that gradient signals can be uninformative, gradients can have vanishing signal-to-noise ratios, and optimization can be ill-conditioned. It provides theoretical bounds based on statistical-query perspectives and gradient variance to explain why gradient-based methods fail on parity/linear-periodic tasks and certain end-to-end decompositions. Through targeted experiments (parity learning, k-rectangles, and piecewise-linear encodings), it demonstrates that architectural choices (notably convolutional designs) and explicit conditioning dramatically improve optimization speed, while non-gradient update rules can overcome limitations posed by flat activations. The work proposes practical remedies—decomposition, conditioning, and forward-only updates—that can extend the robustness of learning algorithms to challenging problem classes, with implications for when and how to apply gradient-based methods in practice.

Abstract

In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.

Figures (8)

• Figure 1: Parity Experiment: Accuracy as a function of the number of training iterations, for various input dimensions.
• Figure 2: Section \ref{['sec:krect']}'s experiment - examples of samples from $X$. The $y$ values of the top and bottom rows are $1$ and $-1$, respectively.
• Figure 3: Performance comparison, Section \ref{['sec:krect']}'s experiment. The red and blue curves correspond to the end-to-end and decomposition approaches, respectively. The plots show the zero-one accuracy with respect to the primary objective, over a held out test set, as a function of training iterations. We have trained the end-to-end network for $20000$ SGD iterations, and the decomposition networks for only $2500$ iterations.
• Figure 4: Section \ref{['sec:krect']}'s experiment: comparing the SNR for the end-to-end approach (red) and the decomposition approach (blue), as a function of $k$, in $\log_e$ scale.
• Figure 5: Examples for decoded outputs of Section \ref{['sec.Piece-wise Linear\n AutoEncoders']}'s experiments, learning to encode PWL curves. In blue are the original curves. In red are the decoded curves. The plot shows the outputs for two curves, after 500, 10000, and 50000 iterations, from left to right.

Theorems & Definitions (9)

• Theorem 1
• Theorem 2: Shamir 2016
• Theorem 3
• Lemma 1
• Lemma 2
• Theorem 4
• Lemma 3
• Lemma 4
• Lemma 5