Puppet-CNN: Input-Adaptive Convolutional Neural Networks with Model Compression using Ordinary Differential Equation

TL;DR

This paper proposes a new CNN framework, named as $\textit{Puppet-CNN}$, which contains two modules: a puppet module and a puppeteer ODE module that can significantly reduce the size of CNN model by only storing and training the parameters of the much smaller puppeteer ODE module.

Abstract

Convolutional Neural Network (CNN) has been applied to more and more scenarios due to its excellent performance in many machine learning tasks, especially with deep and complex structures. However, as the network goes deeper, more parameters need to be stored and optimized. Besides, almost all common CNN models adopt "train-and-use" strategy where the structure is pre-defined and the kernel parameters are fixed after the training with the same structure and set of parameters used for all data without considering the content complexity. In this paper, we propose a new CNN framework, named as $\textit{Puppet-CNN}$, which contains two modules: a $\textit{puppet module}$ and a $\textit{puppeteer module}$. The puppet module is a CNN model used to actually process the input data just like other works, but its depth and kernels are generated by the puppeteer module (realized with Ordinary Differential Equation (ODE)) based on the input complexity each time. By recurrently generating kernel parameters in the puppet module, we can take advantage of the dependence among kernels of different convolutional layers to significantly reduce the size of CNN model by only storing and training the parameters of the much smaller puppeteer ODE module. Through experiments on several datasets, our method has proven to be superior than the traditional CNNs on both performance and efficiency. The model size can be reduced more than 10 times.

Figures (4)

• Figure 1: Comparison among different CNN structures, where the original blocks denote the parameters needed to be stored and optimized. (a) is the traditional CNN where kernel parameters of each layer is independent and needed to be optimized; (b) is the ODE-Net where several layers share the same kernel; (c) is our Puppet-CNN, reducing the actual number of parameters to be optimized while keeping the diversity and relevance among different kernels which are finally applied onto the data.
• Figure 2: The illustration of the weight generation process.
• Figure 3: The illustration of the depth adaption in our method. In the puppet module, the grey convolutional layers with distinct output channels are pre-defined as a template while the red layers are added adaptively according to $dl$.
• Figure 4: Model size comparison between models on different depths.