Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Parallel Training and Transfer Learning for Convolutional Neural Networks by Domain Decomposition

Axel Klawonn, Martin Lanser, Janine Weber

TL;DR

The paper tackles scalable CNN training by adopting a model-parallel, domain-decomposition approach that splits each input image into $N$ subimages processed by smaller local CNNs. Local $K$-class probability outputs from the $N$ submodels are fused by a DNN to produce the global distribution, and the authors compare this fusion to cheaper baselines and to a coherently trained single-model variant, while also exploring transfer learning by pretraining local CNNs and initializing the global model. The transfer-learning strategy pretrains local CNNs for $150$ epochs and uses those weights to initialize the globally trained CNN-DNN, acting as a form of preconditioning. Experiments on CIFAR-10, TF-Flowers, and Chest_CT show that CNN-DNN outperforms average-probability and majority-voting fusion across decompositions, with transfer learning providing additional accuracy gains, though the performance of a coherent end-to-end training depends on the backbone network.

Abstract

Deep convolutional neural networks (CNNs) have been shown to be very successful in a wide range of image processing applications. However, due to their increasing number of model parameters and an increasing availability of large amounts of training data, parallelization strategies to efficiently train complex CNNs are necessary. In previous work by the authors, a novel model parallel CNN architecture was proposed which is loosely inspired by domain decomposition. In particular, the novel network architecture is based on a decomposition of the input data into smaller subimages. For each of these subimages, local CNNs with a proportionally smaller number of parameters are trained in parallel and the resulting local classifications are then aggregated in a second step by a dense feedforward neural network (DNN). In the present work, we compare the resulting CNN-DNN architecture to less costly alternatives to combine the local classifications into a final, global decision. Additionally, we investigate the performance of the CNN-DNN trained as one coherent model as well as using a transfer learning strategy, where the parameters of the pre-trained local CNNs are used as initial values for a subsequently trained global coherent CNN-DNN model.

Model Parallel Training and Transfer Learning for Convolutional Neural Networks by Domain Decomposition

TL;DR

The paper tackles scalable CNN training by adopting a model-parallel, domain-decomposition approach that splits each input image into subimages processed by smaller local CNNs. Local -class probability outputs from the submodels are fused by a DNN to produce the global distribution, and the authors compare this fusion to cheaper baselines and to a coherently trained single-model variant, while also exploring transfer learning by pretraining local CNNs and initializing the global model. The transfer-learning strategy pretrains local CNNs for epochs and uses those weights to initialize the globally trained CNN-DNN, acting as a form of preconditioning. Experiments on CIFAR-10, TF-Flowers, and Chest_CT show that CNN-DNN outperforms average-probability and majority-voting fusion across decompositions, with transfer learning providing additional accuracy gains, though the performance of a coherent end-to-end training depends on the backbone network.

Abstract

Deep convolutional neural networks (CNNs) have been shown to be very successful in a wide range of image processing applications. However, due to their increasing number of model parameters and an increasing availability of large amounts of training data, parallelization strategies to efficiently train complex CNNs are necessary. In previous work by the authors, a novel model parallel CNN architecture was proposed which is loosely inspired by domain decomposition. In particular, the novel network architecture is based on a decomposition of the input data into smaller subimages. For each of these subimages, local CNNs with a proportionally smaller number of parameters are trained in parallel and the resulting local classifications are then aggregated in a second step by a dense feedforward neural network (DNN). In the present work, we compare the resulting CNN-DNN architecture to less costly alternatives to combine the local classifications into a final, global decision. Additionally, we investigate the performance of the CNN-DNN trained as one coherent model as well as using a transfer learning strategy, where the parameters of the pre-trained local CNNs are used as initial values for a subsequently trained global coherent CNN-DNN model.
Paper Structure (7 sections, 3 figures, 2 tables)

This paper contains 7 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Training strategies
  3. Parallel CNN-DNN model architecture
  4. Transfer Learning
  5. Experiments
  6. Network architectures and datasets
  7. Results

Figures (3)

  • Figure 1: Visualization of the CNN-DNN network architecture. Left: The original image is decomposed into $N=4$ nonoverlapping subimages. Middle: The $N=4$ subimages are used as input data for $N$ independent, local CNNs. Right: The probability values of the local CNNs are used as input data for a DNN. The DNN is trained to make a final classification for the decomposed image by weighting the local probability distributions. Figure taken from KLW:DNN-CNN:2023.
  • Figure 2: Left: Exemplary images of the CIFAR-10 dataset Cifar10_TR. Right: Exemplary images of the TF-Flowers dataset tfflowers.
  • Figure 3: Exemplary slices for one chest CT scan taken from the MosMedData dataset Chest_CT.