XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks

TL;DR

This work targets the computational bottleneck of Capsule Networks caused by Dynamic Routing by proposing two fully connected layers, XnODR and XnIDR, which apply Xnorization to linear projections either outside or inside the routing. By inserting these layers into MobileNetV2 and ResNet-50 and evaluating on MNIST, CIFAR-10, and MultiMNIST, the authors demonstrate higher accuracy with fewer parameters and reduced FLOPs compared to baselines. The main contributions are the two novel FC layers, practical integration into common backbones, and comprehensive ablations showing XnIDR generally offers the best accuracy–efficiency trade-off. The results suggest that integrating CapsNet-inspired routing with XNOR-based quantization can yield fast, accurate models suitable for mobile and edge devices, with potential extensions to more scalable datasets and routing methods.

Abstract

Capsule Network is powerful at defining the positional relationship between features in deep neural networks for visual recognition tasks, but it is computationally expensive and not suitable for running on mobile devices. The bottleneck is in the computational complexity of the Dynamic Routing mechanism used between the capsules. On the other hand, XNOR-Net is fast and computationally efficient, though it suffers from low accuracy due to information loss in the binarization process. To address the computational burdens of the Dynamic Routing mechanism, this paper proposes new Fully Connected (FC) layers by xnorizing the linear projection outside or inside the Dynamic Routing within the CapsFC layer. Specifically, our proposed FC layers have two versions, XnODR (Xnorize the Linear Projection Outside Dynamic Routing) and XnIDR (Xnorize the Linear Projection Inside Dynamic Routing). To test the generalization of both XnODR and XnIDR, we insert them into two different networks, MobileNetV2 and ResNet-50. Our experiments on three datasets, MNIST, CIFAR-10, and MultiMNIST validate their effectiveness. The results demonstrate that both XnODR and XnIDR help networks to have high accuracy with lower FLOPs and fewer parameters (e.g., 96.14% correctness with 2.99M parameters and 311.74M FLOPs on CIFAR-10).

Figures (8)

• Figure 1: (a) is the structure of typical CNN-based models; (b) is the structure of models with XnODR/XnIDR. The layers within the light green box are the modified part. We exchange the last convolutional layer and all FC layers with PrimaryCaps layer and XnODR/XnIDR.
• Figure 2: CapsNet Structure. $\hat{I}_{\text{Cap}}$ is the predicted capsules, $I_{\text{Cap}}$ is the updated Capsules, $b$ represents all temporary values, and $v$ represents the activated capsules. Then, $c$ is the coupling coefficient tensor, $c_{ij}$ measures the probability that $\hat{I}_{\text{Cap}_{i}}$ activates $\hat{I}_{\text{Cap}_{j}}$. $N$ is the iteration number.
• Figure 3: Xnorization process. $X$ and $W$ are input tensor and weight variables. $B_{X}$ and $B_{W}$ are binarized sign vectors, while $\alpha_{X}$ and $\alpha_{W}$ are scale factors.
• Figure 4: XnODR (Xnorizes the Linear Projection Outside Dynamic Routing), Version 1 of the proposed Fully Connected layer.
• Figure 5: XnIDR(Xnorize the Linear Projection Inside Dynamic Routing), the Version 2 of proposed Fully Connected layer.