Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightweight Design and Optimization methods for DCNNs: Progress and Futures

Hanhua Long, Wenbin Bi, Jian Sun

TL;DR

This paper surveys lightweight design and optimization methods for DCNNs, addressing manual architectural tuning, AutoML-driven neural architecture search, and model compression techniques. It details strategies such as depthwise separable and group convolutions, structural reparameterization, and various compression methods (pruning, quantization, low-rank, distillation), along with NAS search spaces, strategies, and evaluation approaches. The work highlights current limitations in NAS theory, hardware integration, and interpretability, and proposes directions to broaden applicability beyond image classification and to better harmonize software with edge hardware. Overall, the findings guide practitioners toward integrated, hardware-aware pipelines for building efficient DCNNs suitable for mobile, embedded, and IoT deployments.

Abstract

Lightweight design, as a key approach to mitigate disparity between computational requirements of deep learning models and hardware performance, plays a pivotal role in advancing application of deep learning technologies on mobile and embedded devices, alongside rapid development of smart home, telemedicine, and autonomous driving. With its outstanding feature extracting capabilities, Deep Convolutional Neural Networks (DCNNs) have demonstrated superior performance in computer vision tasks. However, high computational costs and large network architectures severely limit the widespread application of DCNNs on resource-constrained hardware platforms such as smartphones, robots, and IoT devices. This paper reviews lightweight design strategies for DCNNs and examines recent research progress in both lightweight architectural design and model compression. Additionally, this paper discusses current limitations in this field of research and propose prospects for future directions, aiming to provide valuable guidance and reflection for lightweight design philosophy on deep neural networks in the field of computer vision.

Lightweight Design and Optimization methods for DCNNs: Progress and Futures

TL;DR

This paper surveys lightweight design and optimization methods for DCNNs, addressing manual architectural tuning, AutoML-driven neural architecture search, and model compression techniques. It details strategies such as depthwise separable and group convolutions, structural reparameterization, and various compression methods (pruning, quantization, low-rank, distillation), along with NAS search spaces, strategies, and evaluation approaches. The work highlights current limitations in NAS theory, hardware integration, and interpretability, and proposes directions to broaden applicability beyond image classification and to better harmonize software with edge hardware. Overall, the findings guide practitioners toward integrated, hardware-aware pipelines for building efficient DCNNs suitable for mobile, embedded, and IoT deployments.

Abstract

Lightweight design, as a key approach to mitigate disparity between computational requirements of deep learning models and hardware performance, plays a pivotal role in advancing application of deep learning technologies on mobile and embedded devices, alongside rapid development of smart home, telemedicine, and autonomous driving. With its outstanding feature extracting capabilities, Deep Convolutional Neural Networks (DCNNs) have demonstrated superior performance in computer vision tasks. However, high computational costs and large network architectures severely limit the widespread application of DCNNs on resource-constrained hardware platforms such as smartphones, robots, and IoT devices. This paper reviews lightweight design strategies for DCNNs and examines recent research progress in both lightweight architectural design and model compression. Additionally, this paper discusses current limitations in this field of research and propose prospects for future directions, aiming to provide valuable guidance and reflection for lightweight design philosophy on deep neural networks in the field of computer vision.

Paper Structure

This paper contains 20 sections, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Improve Lightweight DCNNs Manually
  3. Adjust Kernel Size
  4. Improve Convolutional Structure
  5. Use Cheaper Operations
  6. Find Equivalent Architecture
  7. Design Lightweight DCNNs Automatically Driven by AutoML
  8. Search Space
  9. Global-based Search Space
  10. Local-based Search Space
  11. Search Strategy
  12. Search Strategy Based on Reinforcement Learning
  13. Gradient-based Search Strategy
  14. Performance Evaluation Strategy
  15. Model Compression for DCNNs
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Depthwise Convolution serves on spatial feature extractor, operating on each channel independently. Meanwhile, Pointwise Convolution acts as a channel mixer, integrating information across various channels.
  • Figure 2: Group Convolution employs a set of kernels, each operating on its respective segment of the input.
  • Figure 3: In RepVGG, a multi-branch structure with residual connection is employed during the training phase, whereas a chain structure equipped with 3×3 kernel is utilized for inference.
  • Figure 4: General Process of Neural Architecture Search.
  • Figure 5: Brief showcase of chain structure and multi-branch structure.
  • ...and 4 more figures