MSPCaps: A Multi-Scale Patchify Capsule Network with Cross-Agreement Routing for Visual Recognition

TL;DR

The paper tackles CapsNet limitations in modeling multi-scale spatial relationships by introducing MSPCaps, a three-part architecture comprising a Multi-Scale ResNet Backbone (MSRB), a Patchify Capsule Layer (PatchifyCaps), and Cross-Agreement Routing (CAR). MSPCaps learns from multi-scale features and fuses them through a progressive, cross-scale voting scheme, enabling scalable Tiny to Large models with strong accuracy and robustness. Extensive experiments on MNIST, FashionMNIST, SVHN, and CIFAR-10 show state-of-the-art performance and improved adversarial resistance, supported by thorough ablations of scale, patch size, and routing strategies. The results demonstrate that structured, patch-based multi-scale capsules with adaptive cross-scale routing yield efficient and robust feature representations for visual recognition.

Abstract

Capsule Network (CapsNet) has demonstrated significant potential in visual recognition by capturing spatial relationships and part-whole hierarchies for learning equivariant feature representations. However, existing CapsNet and variants often rely on a single high-level feature map, overlooking the rich complementary information from multi-scale features. Furthermore, conventional feature fusion strategies (e.g., addition and concatenation) struggle to reconcile multi-scale feature discrepancies, leading to suboptimal classification performance. To address these limitations, we propose the Multi-Scale Patchify Capsule Network (MSPCaps), a novel architecture that integrates multi-scale feature learning and efficient capsule routing. Specifically, MSPCaps consists of three key components: a Multi-Scale ResNet Backbone (MSRB), a Patchify Capsule Layer (PatchifyCaps), and Cross-Agreement Routing (CAR) blocks. First, the MSRB extracts diverse multi-scale feature representations from input images, preserving both fine-grained details and global contextual information. Second, the PatchifyCaps partitions these multi-scale features into primary capsules using a uniform patch size, equipping the model with the ability to learn from diverse receptive fields. Finally, the CAR block adaptively routes the multi-scale capsules by identifying cross-scale prediction pairs with maximum agreement. Unlike the simple concatenation of multiple self-routing blocks, CAR ensures that only the most coherent capsules contribute to the final voting. Our proposed MSPCaps achieves remarkable scalability and superior robustness, consistently surpassing multiple baseline methods in terms of classification accuracy, with configurations ranging from a highly efficient Tiny model (344.3K parameters) to a powerful Large model (10.9M parameters), highlighting its potential in advancing feature representation learning.

Figures (4)

• Figure 1: Comparative overview of conventional CapsNet and our proposed MSPCaps. The path (a) $\rightarrow$ (b) shows the conventional CapsNet with dynamic routing, where the prediction capsules $\hat{u}$ are first generated by applying weight matrices $W$ on primary capsules $u$, followed by an iterative routing-by-agreement process to update the coupling coefficients $C$ for $n$ iterations. The class capsules are then produced in the last iteration based on output capsules $v$. In contrast, the path (a) $\rightarrow$ (c) shows the proposed MSPCaps, which leverages multi-scale features to generate initial capsules and employs a non-iterative cross-agreement routing mechanism for voting to compute output capsules.
• Figure 2: (a) The overall architecture of MSPCaps, which consists of an MSRB, a PatchifyCaps, and two CAR blocks. The backbone is designed with three residual blocks to generate features at varying scales. (b) Each residual block starts with a downsampling convolution (stride=2), followed by $n$ identical convolutional modules, each with a residual connection.
• Figure 3: An illustration of the cross-agreement routing block, where $A$ represents the raw agreement scores matrix and $C$ denotes the final coupling coefficients matrix.
• Figure 4: Classification accuracy of MSPCaps-L, MSPCaps-T, and CapsNet against FGSM and BIM adversarial attacks on CIFAR-10, plotted with attack strength $\epsilon$.