Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training

Lorenzo Baraldi, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Andrea Pilzer, Rita Cucchiara

TL;DR

The paper tackles the challenge of noisy input and weak cross-phase alignment in Masked Image Modeling for Vision Transformers. It proposes MaPeT, a masked-and-permuted pre-training framework that uses a two-stream attention mechanism and position-aware masked tokens to capture intra-patch dependencies while preserving access to full patch sequences. A key contribution is the k-CLIP tokenizer, which discretizes CLIP features into 8192 tokens to provide semantically rich supervision without dataset-specific autoencoders. Empirical results on ImageNet-1k and ADE20K demonstrate competitive or superior performance to existing self-supervised methods under fair settings, highlighting the practical impact of improved pre-training objectives and tokenization strategies. The work also includes extensive analyses of tokenizer quality and cross-domain transfer, underscoring MaPeT’s potential for robust, generalizable representations in vision tasks.

Abstract

The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of many different visual tasks. In this context, recent approaches have employed the Masked Image Modeling paradigm, which pre-trains a backbone by reconstructing visual tokens associated with randomly masked image patches. This masking approach, however, introduces noise into the input data during pre-training, leading to discrepancies that can impair performance during the fine-tuning phase. Furthermore, input masking neglects the dependencies between corrupted patches, increasing the inconsistencies observed in downstream fine-tuning tasks. To overcome these issues, we propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT), that employs autoregressive and permuted predictions to capture intra-patch dependencies. In addition, MaPeT employs auxiliary positional information to reduce the disparity between the pre-training and fine-tuning phases. In our experiments, we employ a fair setting to ensure reliable and meaningful comparisons and conduct investigations on multiple visual tokenizers, including our proposed $k$-CLIP which directly employs discretized CLIP features. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting. We release an implementation of our code and models at https://github.com/aimagelab/MaPeT.

Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training

TL;DR

The paper tackles the challenge of noisy input and weak cross-phase alignment in Masked Image Modeling for Vision Transformers. It proposes MaPeT, a masked-and-permuted pre-training framework that uses a two-stream attention mechanism and position-aware masked tokens to capture intra-patch dependencies while preserving access to full patch sequences. A key contribution is the k-CLIP tokenizer, which discretizes CLIP features into 8192 tokens to provide semantically rich supervision without dataset-specific autoencoders. Empirical results on ImageNet-1k and ADE20K demonstrate competitive or superior performance to existing self-supervised methods under fair settings, highlighting the practical impact of improved pre-training objectives and tokenization strategies. The work also includes extensive analyses of tokenizer quality and cross-domain transfer, underscoring MaPeT’s potential for robust, generalizable representations in vision tasks.

Abstract

The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of many different visual tasks. In this context, recent approaches have employed the Masked Image Modeling paradigm, which pre-trains a backbone by reconstructing visual tokens associated with randomly masked image patches. This masking approach, however, introduces noise into the input data during pre-training, leading to discrepancies that can impair performance during the fine-tuning phase. Furthermore, input masking neglects the dependencies between corrupted patches, increasing the inconsistencies observed in downstream fine-tuning tasks. To overcome these issues, we propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT), that employs autoregressive and permuted predictions to capture intra-patch dependencies. In addition, MaPeT employs auxiliary positional information to reduce the disparity between the pre-training and fine-tuning phases. In our experiments, we employ a fair setting to ensure reliable and meaningful comparisons and conduct investigations on multiple visual tokenizers, including our proposed -CLIP which directly employs discretized CLIP features. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting. We release an implementation of our code and models at https://github.com/aimagelab/MaPeT.
Paper Structure (13 sections, 3 equations, 5 figures, 10 tables)

This paper contains 13 sections, 3 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Proposed Method
  5. Masked and Permuted Pre-Training
  6. $k$-CLIP: Discretized CLIP-based Tokenizer
  7. Experiments
  8. Experimental Setup
  9. Experimental Results
  10. Visual Tokenizer Analysis
  11. Conclusion
  12. Additional Codebook Visualization
  13. Additional Implementation Details

Figures (5)

  • Figure 1: (a) Masked Image Modeling (MIM). (b) Permuted Image Modeling (PIM). (c) Masked and Permuted pre-training for Vision Transformers (MaPeT). While MIM reconstructs visual tokens from randomly masked image patches, PIM autoregressively predicts tokens associated with permuted image patches. MaPeT uses PIM to capture intra-patch dependency and takes auxiliary position information as input to ensure that the model sees a full sequence of patches at each target position.
  • Figure 2: Overview of our MaPeT pre-training (a), with content stream attention (b), which follows the standard self-attention mechanism, and query stream attention (c), which lacks information about the content of the patch embedding $\bm{x}_{z_t}$ whose visual token $v_{z_t}$ is to be predicted. The blue and green masks in (a) are the content and query attention masks employed in the two-stream self-attention.
  • Figure 3: Visualization of image patches corresponding to the discrete tokens of our $k$-CLIP codebook, where semantically similar patches consistently share the same discrete token.
  • Figure 4: Visualization of image patches mismatching the semantic concept associated with the discrete tokens contained within our $k$-CLIP codebook. Corresponding image patches are marked in red rectangle.
  • Figure 5: Visualization of image patches corresponding to the discrete tokens contained within our $k$-CLIP codebook. The codebook exhibits a high degree of semantic density and coherency, i.e., semantically similar image patches are consistently linked with the same discrete token in the visual codebook. Corresponding image patches are marked in red rectangle.