Enhanced Convolutional Neural Tangent Kernels

TL;DR

The paper targets improving CNTK/CNN-GP kernels for image classification by embedding data augmentation into the kernel via Local Average Pooling (LAP) and by enriching input representations with random-patch pre-processing. LAP introduces a localized translation symmetry in the kernel recursions and is shown to correspond to GAP under certain padding, while staying computationally efficient. Empirically, LAP improves CIFAR-10 and Fashion-MNIST performance; when combined with horizontal flip augmentation and Coates-style random patches, CIFAR-10 accuracy reaches 89%, matching AlexNet without training. These results illustrate that fixed-kernel methods can capture practical deep-learning ideas and yield strong, train-free classifiers.

Abstract

Recent research shows that for training with $\ell_2$ loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012). Note that this is the best such result we know of for a classifier that is not a trained neural network. Similar improvements are obtained for Fashion-MNIST.

Figures (2)

• Figure 1: Randomly sampled images with full translation data augmentation and local translation data augmentation from CIFAR-10. Full translation data augmentation can create unrealistic images that harm the performance whereas local translation data augmentation creates more realistic images.
• Figure 2: Test accuracy of 10-layer CNN with various values for the $c$ parameter in $\textsf{BBlur}$.

Theorems & Definitions (6)

• Definition 4.1
• Theorem 4.1
• Corollary 4.1
• Corollary 4.2
• Definition B.1: Group of Operators
• proof : Proof of Theorem \ref{['thm:gap_equivalence']}