Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

Gaurav Menghani

TL;DR

This survey identifies the urgent need to optimize deep learning beyond accuracy by prioritizing efficiency across inference and training footprints. It introduces a five-domain mental model—Compression, Learning, Automation, Efficient Architectures, and Infrastructure—and surveys core techniques in each, including pruning, quantization, distillation, SSL, NAS, and specialized hardware. The authors provide an experiment-based guide, demonstrating practical tradeoffs on CIFAR-10 with Shrink/Improve and Grow/Improve/Shrink strategies, and supply code to help practitioners deploy pareto-optimal models. The work synthesizes modeling, tooling, and hardware advances to enable scalable, on-device, and energy-efficient AI, offering a concrete framework for both researchers and engineers to pursue efficiency-frontier innovations.

Abstract

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

TL;DR

This survey identifies the urgent need to optimize deep learning beyond accuracy by prioritizing efficiency across inference and training footprints. It introduces a five-domain mental model—Compression, Learning, Automation, Efficient Architectures, and Infrastructure—and surveys core techniques in each, including pruning, quantization, distillation, SSL, NAS, and specialized hardware. The authors provide an experiment-based guide, demonstrating practical tradeoffs on CIFAR-10 with Shrink/Improve and Grow/Improve/Shrink strategies, and supply code to help practitioners deploy pareto-optimal models. The work synthesizes modeling, tooling, and hardware advances to enable scalable, on-device, and energy-efficient AI, offering a concrete framework for both researchers and engineers to pursue efficiency-frontier innovations.

Abstract

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Paper Structure

This paper contains 28 sections, 8 equations, 27 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Efficient Deep Learning
  3. A Mental Model
  4. Landscape of Efficient Deep Learning
  5. Compression Techniques
  6. Pruning
  7. Quantization
  8. Other Compression Techniques
  9. Learning Techniques
  10. Distillation
  11. Data Augmentation
  12. Self-Supervised Learning
  13. Automation
  14. Hyper-Parameter Optimization (HPO)
  15. Neural Architecture Search (NAS)
  16. ...and 13 more sections

Figures (27)

  • Figure 1: Growth in the number of parameters in Computer Vision models over time. PapersWithCode2021
  • Figure 2: Pareto Optimality: Green dots represent pareto-optimal models (together forming the pareto-frontier), where none of the other models (red dots) get better accuracy with the same inference latency, or the other way around.
  • Figure 3: A mental model for thinking about algorithms, techniques, and tools related to efficiency in Deep Learning.
  • Figure 4: A simplified illustration of pruning weights (connections) and neurons (nodes) in a neural network comprising of fully connected layers.
  • Figure 5: Quantizing floating-point continuous values to discrete fixed-point values. The continuous values are clamped to the range $x_{min}$ to $x_{max}$, and are mapped to discrete values in [$0$, $2^b - 1$] (in the above figure, $b = 3$, hence the quantized values are in the range [$0, 7$].
  • ...and 22 more figures