Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thoughts on Objectives of Sparse and Hierarchical Masked Image Model

Asahi Miyazaki, Tsuyoshi Okita

TL;DR

This work investigates how masking pattern choices in masked image modeling affect pretraining when using SparK's sparse, hierarchical backbone. It introduces Mesh Mask as an alternative to common masking patterns and evaluates its impact on downstream brain CT tumor detection. Across experiments, SparK generally outperforms SimCLR and MFF, with patch-level masking (random and mesh) yielding stronger results than block-level schemes, though mesh masking remains competitive with random masking and is sensitive to data augmentation. The findings guide masking-pattern design in MIM, with practical implications for medical-imaging pretraining and representation learning.

Abstract

Masked image modeling is one of the most poplular objectives of training. Recently, the SparK model has been proposed with superior performance among self-supervised learning models. This paper proposes a new mask pattern for this SparK model, proposing it as the Mesh Mask-ed SparK model. We report the effect of the mask pattern used for image masking in pre-training on performance.

Thoughts on Objectives of Sparse and Hierarchical Masked Image Model

TL;DR

This work investigates how masking pattern choices in masked image modeling affect pretraining when using SparK's sparse, hierarchical backbone. It introduces Mesh Mask as an alternative to common masking patterns and evaluates its impact on downstream brain CT tumor detection. Across experiments, SparK generally outperforms SimCLR and MFF, with patch-level masking (random and mesh) yielding stronger results than block-level schemes, though mesh masking remains competitive with random masking and is sensitive to data augmentation. The findings guide masking-pattern design in MIM, with practical implications for medical-imaging pretraining and representation learning.

Abstract

Masked image modeling is one of the most poplular objectives of training. Recently, the SparK model has been proposed with superior performance among self-supervised learning models. This paper proposes a new mask pattern for this SparK model, proposing it as the Mesh Mask-ed SparK model. We report the effect of the mask pattern used for image masking in pre-training on performance.
Paper Structure (21 sections, 9 equations, 6 figures, 9 tables, 1 algorithm)

This paper contains 21 sections, 9 equations, 6 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. SparK
  4. Masking Method of MIM
  5. Experiment Setting of Self-Supervised Learning
  6. Datasets
  7. Metrics for Performance Evaluation
  8. Environment
  9. Data Augmentation
  10. Class Weighting
  11. Comparison of Self-Supervised Learning Methods
  12. SimCLR
  13. MFF
  14. SparK
  15. Results
  16. ...and 6 more sections

Figures (6)

  • Figure 1: F1 Scores for SimCLR
  • Figure 2: F1 Scores for MFF
  • Figure 3: F1 Scores for SparK
  • Figure 4: Two possible candidates for non-masked patches used in the mesh mask. The blue semi-transparent patches represent the candidates for non-masked patches.
  • Figure 5:
  • ...and 1 more figures