Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

Zhigang Wang, Hangyu Yang, Ning Wang, Chuanfei Xu, Jie Nie, Zhiqiang Wei, Yu Gu, Ge Yu

TL;DR

The paper tackles the memory bottleneck in CNN training by rethinking dataflow from layer-centric to row-centric, enabling reuse of memory across layers and earlier release of intermediate data. It introduces LR-CNN with two row-partitioning strategies, Two-Phase Sharing (2PS) and Overlapping Partitioning (OverL), and shows how to bound memory via distinct FP/BP granularities and, optionally, checkpointing. Empirical results on VGG-16 and ResNet-50 demonstrate substantial memory reductions (up to around 78% in baseline comparisons and larger gains with hybrids) with acceptable latency and preserved convergence. The approach is orthogonal to and compatible with existing memory-reduction techniques, offering a lightweight, hardware-free path to scalable CNN training on memory-limited accelerators.

Abstract

In the last decade, Convolutional Neural Network with a multi-layer architecture has advanced rapidly. However, training its complex network is very space-consuming, since a lot of intermediate data are preserved across layers, especially when processing high-dimension inputs with a big batch size. That poses great challenges to the limited memory capacity of current accelerators (e.g., GPUs). Existing efforts mitigate such bottleneck by external auxiliary solutions with additional hardware costs, and internal modifications with potential accuracy penalty. Differently, our analysis reveals that computations intra- and inter-layers exhibit the spatial-temporal weak dependency and even complete independency features. That inspires us to break the traditional layer-by-layer (column) dataflow rule. Now operations are novelly re-organized into rows throughout all convolution layers. This lightweight design allows a majority of intermediate data to be removed without any loss of accuracy. We particularly study the weak dependency between two consecutive rows. For the resulting skewed memory consumption, we give two solutions with different favorite scenarios. Evaluations on two representative networks confirm the effectiveness. We also validate that our middle dataflow optimization can be smoothly embraced by existing works for better memory reduction.

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

TL;DR

The paper tackles the memory bottleneck in CNN training by rethinking dataflow from layer-centric to row-centric, enabling reuse of memory across layers and earlier release of intermediate data. It introduces LR-CNN with two row-partitioning strategies, Two-Phase Sharing (2PS) and Overlapping Partitioning (OverL), and shows how to bound memory via distinct FP/BP granularities and, optionally, checkpointing. Empirical results on VGG-16 and ResNet-50 demonstrate substantial memory reductions (up to around 78% in baseline comparisons and larger gains with hybrids) with acceptable latency and preserved convergence. The approach is orthogonal to and compatible with existing memory-reduction techniques, offering a lightweight, hardware-free path to scalable CNN training on memory-limited accelerators.

Abstract

In the last decade, Convolutional Neural Network with a multi-layer architecture has advanced rapidly. However, training its complex network is very space-consuming, since a lot of intermediate data are preserved across layers, especially when processing high-dimension inputs with a big batch size. That poses great challenges to the limited memory capacity of current accelerators (e.g., GPUs). Existing efforts mitigate such bottleneck by external auxiliary solutions with additional hardware costs, and internal modifications with potential accuracy penalty. Differently, our analysis reveals that computations intra- and inter-layers exhibit the spatial-temporal weak dependency and even complete independency features. That inspires us to break the traditional layer-by-layer (column) dataflow rule. Now operations are novelly re-organized into rows throughout all convolution layers. This lightweight design allows a majority of intermediate data to be removed without any loss of accuracy. We particularly study the weak dependency between two consecutive rows. For the resulting skewed memory consumption, we give two solutions with different favorite scenarios. Evaluations on two representative networks confirm the effectiveness. We also validate that our middle dataflow optimization can be smoothly embraced by existing works for better memory reduction.
Paper Structure (20 sections, 19 equations, 11 figures, 1 table, 1 algorithm)

This paper contains 20 sections, 19 equations, 11 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Convolutional Neural Network (CNN)
  4. Space Complexity of Training CNN
  5. LR-CNN: Lightweight Row-centric CNN Training
  6. Overview Design
  7. Convergence Analysis
  8. Customizing Row Granularity in FP and BP
  9. Row Partitioning with Complex-objectives
  10. Two-phase Partitioning by Sharing Inter-row Computations
  11. Overlapping Partitioning for Independent Computations
  12. EXPERIMENTS
  13. Experimental Setups
  14. Overall Performance Comparison
  15. Evaluation on Scalability
  16. ...and 5 more sections

Figures (11)

  • Figure 1: Traditional CNN
  • Figure 2: Row-centric CNN
  • Figure 3: Variation of feature map
  • Figure 4: Illustration of skewness in 2PS
  • Figure 5: Illustration of overlapping partitioning
  • ...and 6 more figures