A Survey of Deep Learning: From Activations to Transformers

TL;DR

This survey provides a holistic overview of deep learning progress across learning objectives, architectures, and training techniques, with emphasis on activations, normalization, skip connections, attention, transformers, and graph neural networks. It adopts a cross-disciplinary perspective and leverages leaderboards to identify influential works since 2016, while also highlighting rising stars. The authors distill key design patterns such as Multi-X, higher-order layers, moving averages, decomposition, and weighted inputs, and discuss commercially deployed models like GPT-4 and PaLM 2. They argue for more radical, non-incremental innovations to drive future progress and offer a curated map to connect diverse subfields.

Abstract

Deep learning has made tremendous progress in the last decade. A key success factor is the large amount of architectures, layers, objectives, and optimization techniques. They include a myriad of variants related to attention, normalization, skip connections, transformers and self-supervised learning schemes -- to name a few. We provide a comprehensive overview of the most important, recent works in these areas to those who already have a basic understanding of deep learning. We hope that a holistic and unified treatment of influential, recent works helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for many of the successful innovations over the last decade as well as works that can be seen as rising stars. We also include a discussion on recent commercially built, closed-source models such as OpenAI's GPT-4 and Google's PaLM 2.

Figures (2)

• Figure 1: Categorization of deep learning and areas covered in the survey
• Figure 2: Transformer with the four basic blocks on top and the encoder and decoder at the bottom