Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Canvas: End-to-End Kernel Architecture Search in Neural Networks

Chenggang Zhao, Genghan Zhang, Ao Shen, Mingyu Gao

TL;DR

An end-to-end framework, Canvas, to find high-quality kernels as convolution replacements and verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations.

Abstract

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations. For source code and implementation, we open-sourced Canvas at https://github.com/tsinghua-ideal/Canvas.

Canvas: End-to-End Kernel Architecture Search in Neural Networks

TL;DR

An end-to-end framework, Canvas, to find high-quality kernels as convolution replacements and verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations.

Abstract

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations. For source code and implementation, we open-sourced Canvas at https://github.com/tsinghua-ideal/Canvas.
Paper Structure (17 sections, 1 theorem, 4 equations, 9 figures, 1 table)

This paper contains 17 sections, 1 theorem, 4 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Kernel Architecture Search
  4. Canvas Design Overview
  5. Fine-Grained Primitives
  6. Kernel Generation
  7. Sampling Algorithm
  8. Dynamic Variable Solver by Shape Matching
  9. Dynamic Variable Solver by Analytical Constraints
  10. Kernel Evaluation
  11. Results
  12. Experimental Setups
  13. End-to-End Performance
  14. Model Size Reduction
  15. Kernel-Level Analysis of ResNet-34
  16. ...and 2 more sections

Key Result

Theorem 1

For any tensor constructed in Canvas where all its dimensions are in the form of $D=\frac{\text{numerator}}{\text{denominator}}$ (e.g. $D=\frac{C}{G}$ or $D=CK_H$), there is

Figures (9)

  • Figure 1: System overview of Canvas and its workflow. The micro-DAG random sampler generates a kernel that contains two dynamic variables, $x_1$ and $x_2$. The solver by shape matching sets the value of $x_2$ when merging the two branches into one. The solver by constraints assigns concrete values to the remaining $x_1$ for each replacement target. The code generator implements the resultant kernels. Finally, the evaluator selects the best candidate according to accuracy and runtime measurements.
  • Figure 2: A single kernel template of $[C, H, W]$$\Rightarrow$$[C, H, W]$ is applied to general convolutions with different $C_{\text{in}}$ and $C_{\text{out}}$ values.
  • Figure 3: An example of solving dynamic variables for different convolution targets by the constraint solver.
  • Figure 4: End-to-end performance comparison between Canvas and the baselines.
  • Figure 5: Model size comparison between the original size and that discovered by Canvas, for both the entire network and the feature extraction front-end excluding the final classifiers. Each number indicates the ratio of the new model size compared to the original size.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 1