AutoMC: Automated Model Compression based on Domain Knowledge and Progressive search strategy

TL;DR

AutoMC addresses the challenge of automatically designing compression schemes for neural networks by composing multiple existing compression methods under a domain-knowledge framework. It introduces a domain-knowledge graph and an experimental-experience embedding, combined with a progressive, Pareto-aware search that expands only promising next steps to locate high-quality schemes. Empirical results show AutoMC outperforms hand-designed methods and standard AutoML/NAS approaches, with transferable compression strategies across architectures. The approach significantly lowers the expertise and time required to deploy compact, accurate models in resource-constrained settings.

Abstract

Model compression methods can reduce model complexity on the premise of maintaining acceptable performance, and thus promote the application of deep neural networks under resource constrained environments. Despite their great success, the selection of suitable compression methods and design of details of the compression scheme are difficult, requiring lots of domain knowledge as support, which is not friendly to non-expert users. To make more users easily access to the model compression scheme that best meet their needs, in this paper, we propose AutoMC, an effective automatic tool for model compression. AutoMC builds the domain knowledge on model compression to deeply understand the characteristics and advantages of each compression method under different settings. In addition, it presents a progressive search strategy to efficiently explore pareto optimal compression scheme according to the learned prior knowledge combined with the historical evaluation information. Extensive experimental results show that AutoMC can provide satisfying compression schemes within short time, demonstrating the effectiveness of AutoMC.

Figures (6)

• Figure 1: AutoMC's search space can be described in a tree structure. Each node has 4,525 children nodes, corresponding to the 4,525 compression strategies in Table \ref{['table1']}.
• Figure 2: The structure of knowledge graph and $\mathcal{NN}_{exp}$ that are used for embedding learning. $S_{i,j}$ is the setting of hyperparameter $HP_{i}$.
• Figure 3: Structure of $\mathcal{F}_{mo}$. The embedding of $s_i$ and $s^{\ast}$ are provided by Algorithm \ref{['alg1']}.
• Figure 4: Pareto optimal results searched by different AutoML algorithms on Exp1 and Exp2.
• Figure 5: Pareto optimal results serach by different versions of AutoMC on Exp1 and Exp2.