Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, Jiashi Feng

TL;DR

Vision Permutator addresses data efficiency in MLP-like vision models by preserving 2D spatial information through separate height- and width-wise encoding (Permute-MLP) and channel mixing (Channel-MLP) on 3D token representations. The architecture stacks Permutator blocks, enabling position-sensitive feature extraction without convolutions or self-attention and utilizing a fusion-then-FC final stage. On ImageNet-1k without extra data, it achieves strong results (81.5% with 25M params, up to 83.2% with 88M) and shows clear gains from diverse ablations, including fine-level token representations and weighted fusion. The work demonstrates that explicitly modeling 2D spatial structure in MLP-like models can rival CNNs and ViTs under similar resource constraints, while highlighting scalability and down-stream task challenges due to fixed input shapes.

Abstract

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

TL;DR

Vision Permutator addresses data efficiency in MLP-like vision models by preserving 2D spatial information through separate height- and width-wise encoding (Permute-MLP) and channel mixing (Channel-MLP) on 3D token representations. The architecture stacks Permutator blocks, enabling position-sensitive feature extraction without convolutions or self-attention and utilizing a fusion-then-FC final stage. On ImageNet-1k without extra data, it achieves strong results (81.5% with 25M params, up to 83.2% with 88M) and shows clear gains from diverse ablations, including fine-level token representations and weighted fusion. The work demonstrates that explicitly modeling 2D spatial structure in MLP-like models can rival CNNs and ViTs under similar resource constraints, while highlighting scalability and down-stream task challenges due to fixed input shapes.

Abstract

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.

Paper Structure

This paper contains 9 sections, 2 equations, 2 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Permutator
  4. Permutator
  5. Various Configurations of Vision Permutator
  6. Experiments
  7. Experiment Setup
  8. Main Results on ImageNet
  9. Ablation Analysis

Figures (2)

  • Figure 1: Basic architecture of the proposed Vision Permutator. The evenly divided image patches are tokenized with linear projection first and then fed into a sequence of Permutators for feature encoding. A global average pooling layer followed by a fully-connected layer is finally used to predict the class.
  • Figure 2: Basic structure of the proposed Permute-MLP layer. The proposed Permute-MLP layer contains three branches that are responsible for encoding features along the height, width, and channel dimensions, respectively. The outputs from the three branches are then combined using element-wise addition, followed by a fully-connected layer for feature fusion.