NeoNeXt: Novel neural network operator and architecture based on the patch-wise matrix multiplications

TL;DR

This paper proposes a novel foundation operation - NeoCell - which learns matrix patterns and performs patchwise matrix multiplications with the input data and validate NeoNeXt family of models based on this operation on ImageNet-1K classification task and show that they achieve competitive quality.

Abstract

Most of the computer vision architectures nowadays are built upon the well-known foundation operations: fully-connected layers, convolutions and multi-head self-attention blocks. In this paper we propose a novel foundation operation - NeoCell - which learns matrix patterns and performs patchwise matrix multiplications with the input data. The main advantages of the proposed operator are (1) simple implementation without need in operations like im2col, (2) low computational complexity (especially for large matrices) and (3) simple and flexible implementation of up-/down-sampling. We validate NeoNeXt family of models based on this operation on ImageNet-1K classification task and show that they achieve competitive quality.

Figures (6)

• Figure 1: ImageNet-1K accuracy vs parameters. NeoNeXt is compared to ResNet, DeiT and ConvNeXt. Alough we do not achieve state-of-the-art performance, we prove that NeoCell operation is a viable choice. It's also shown that NeoNeXt quality grows with the number of parameters, which means that models are scalable. IN1K means that ConvNeXt was not pretrained on external data such as ImageNet-22K.
• Figure 2: Illustration of splitting of the data into the patches
• Figure 3: Methods of spatial size change
• Figure 4: Methods of inter-patch information exchange. Only right NeoCell matrices are shown here.
• Figure 5: Examples of NeoInit method for different matrices