Analytic Convolutional Layer: A Step to Analytic Neural Network

TL;DR

The paper addresses the data-driven parameter explosion and limited interpretability of traditional CNNs by proposing the Analytic Convolutional Layer (ACL), a model-driven layer that blends Analytic Kernel Functions (ACKs) with plain kernels. AKPs parameterize each analytic kernel, enabling adaptive representation of feature spaces while maintaining a compact parameter budget; ACLs can also approximate pretrained kernels with far fewer parameters, supporting compression and interpretability. Experiments across datasets (Oxford Flowers, MNIST, Food-101, CIFAR-10) demonstrate that ACLs achieve competitive accuracy with substantial parameter reductions, and AnaNNs (Analytic Neural Networks) show improved explainability and maintain performance gains in several configurations. The work establishes a foundation for analytic neural networks, enabling principled integration of prior knowledge, interpretability, and potential efficiency advantages, with public code to foster replication and further exploration.

Abstract

The prevailing approach to embedding prior knowledge within convolutional layers typically includes the design of steerable kernels or their modulation using designated kernel banks. In this study, we introduce the Analytic Convolutional Layer (ACL), an innovative model-driven convolutional layer, which is a mosaic of analytical convolution kernels (ACKs) and traditional convolution kernels. ACKs are characterized by mathematical functions governed by analytic kernel parameters (AKPs) learned in training process. Learnable AKPs permit the adaptive update of incorporated knowledge to align with the features representation of data. Our extensive experiments demonstrate that the ACLs not only have a remarkable capacity for feature representation with a reduced number of parameters but also attain increased reliability through the analytical formulation of ACKs. Furthermore, ACLs offer a means for neural network interpretation, thereby paving the way for the intrinsic interpretability of neural network. The source code will be published in company with the paper.

Figures (6)

• Figure 1: Convolution kernels pretrained on ImageNet exhibit discernible patterns, which we demonstrate can be analytically modeled using mathematically-defined kernels. The figure illustrates the first layer kernels from pretrained AlexNet krizhevsky2012imagenet and ResNet-50 He2015DeepRL on the left and right, respectively, with the corresponding analytical kernels in the center. It is important to note that these vibrant kernels represent combinations of three distinct kernels within the red, green, and blue channels, respectively.
• Figure 2: The diagram illustrates the workflow of the ACL and delineates the process of transforming a standard neural network into an AnaNN (Analytic Neural Network) to enhance its explainability. The parameters enclosed within the orange dashed box, which include plain kernels and AKPs, are the only ones that are learned and updated. The blue dashed box represents the typical volume of parameters that must be learned and updated in traditional convolutional layers.
• Figure 3: Certain patterns observed in pretrained convolution kernels can be accurately modeled using ACKs. This demonstration illustrates the ability to use significantly fewer AKPs to approximate pretrained kernels. On the left, there is a pattern (which is actually a combination of three kernels) selected from the pretrained weights of ResNet-50. It is precisely modeled by three Gabor kernels, described by a mere 12 AKP values.
• Figure 4: (a): Visualization of an ACL mosaic in its initialization state with the arrangement $(3\times 64)\mathit{G}_{30}\mathit{Lg}_{15}\mathit{Lt}_{15}\mathit{Tf}_{30}\mathit{Ts}_{6}\mathit{P}_{96}$. The kernels framed by red, purple, pink, blue, green, and orange outlines correspond to Gabor, LoG, LoT, TGD1st, TGD2nd, and Plain kernels, respectively. ACKs are initialized by random AKP values, while Plain kernels are initialized with pretrained kernels. They are reorganized into a $64 \times 3$ shape for display in RGB mode. (b): Top-1 accuracy contributions explanation of the various kernel types in exp3 elucidated by an ablation study.
• Figure 5: a: the structure of AnaNN-LeNet. b: our four-layer AnaNN. Other network modules omitted for simplicity.

Theorems & Definitions (1)

• definition 1