Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Self Organizing Map with Visual Transformer

Alan Luo, Kaiwen Yuan

TL;DR

This work tackles the data-hungry nature of Vision Transformers by introducing ViT-SOM, a framework that pairs a compact Vision Transformer with a Self-Organizing Map to impose topology-inspired inductive biases on high-dimensional embeddings. It optimizes a joint loss $L_{total} = L_{nn} + \gamma \cdot L_{som}$, balancing neural feature learning with topological organization, and uses a cosine-based topological distance $d_{ij}(z) = 1 - \frac{z \cdot w_{ij}}{\|z\|\|w_{ij}\|}$ to align embeddings with SOM prototypes. Empirically, ViT-SOM improves unsupervised clustering purity over SOM-VAE and DESOM and achieves state-of-the-art-from-scratch accuracy on several small to mid-size datasets with far fewer parameters, validating the mutual benefits of integrating ViTs with SOMs. The results imply practical utility for data-limited vision tasks and provide a foundation for further exploration of topology-aware deep learning, with code released for reproducibility.

Abstract

Vision Transformers (ViTs) have demonstrated exceptional performance in various vision tasks. However, they tend to underperform on smaller datasets due to their inherent lack of inductive biases. Current approaches address this limitation implicitly-often by pairing ViTs with pretext tasks or by distilling knowledge from convolutional neural networks (CNNs) to strengthen the prior. In contrast, Self-Organizing Maps (SOMs), a widely adopted self-supervised framework, are inherently structured to preserve topology and spatial organization, making them a promising candidate to directly address the limitations of ViTs in limited or small training datasets. Despite this potential, equipping SOMs with modern deep learning architectures remains largely unexplored. In this study, we conduct a novel exploration on how Vision Transformers (ViTs) and Self-Organizing Maps (SOMs) can empower each other, aiming to bridge this critical research gap. Our findings demonstrate that these architectures can synergistically enhance each other, leading to significantly improved performance in both unsupervised and supervised tasks. Code is publicly available on GitHub.

Simple Self Organizing Map with Visual Transformer

TL;DR

This work tackles the data-hungry nature of Vision Transformers by introducing ViT-SOM, a framework that pairs a compact Vision Transformer with a Self-Organizing Map to impose topology-inspired inductive biases on high-dimensional embeddings. It optimizes a joint loss , balancing neural feature learning with topological organization, and uses a cosine-based topological distance to align embeddings with SOM prototypes. Empirically, ViT-SOM improves unsupervised clustering purity over SOM-VAE and DESOM and achieves state-of-the-art-from-scratch accuracy on several small to mid-size datasets with far fewer parameters, validating the mutual benefits of integrating ViTs with SOMs. The results imply practical utility for data-limited vision tasks and provide a foundation for further exploration of topology-aware deep learning, with code released for reproducibility.

Abstract

Vision Transformers (ViTs) have demonstrated exceptional performance in various vision tasks. However, they tend to underperform on smaller datasets due to their inherent lack of inductive biases. Current approaches address this limitation implicitly-often by pairing ViTs with pretext tasks or by distilling knowledge from convolutional neural networks (CNNs) to strengthen the prior. In contrast, Self-Organizing Maps (SOMs), a widely adopted self-supervised framework, are inherently structured to preserve topology and spatial organization, making them a promising candidate to directly address the limitations of ViTs in limited or small training datasets. Despite this potential, equipping SOMs with modern deep learning architectures remains largely unexplored. In this study, we conduct a novel exploration on how Vision Transformers (ViTs) and Self-Organizing Maps (SOMs) can empower each other, aiming to bridge this critical research gap. Our findings demonstrate that these architectures can synergistically enhance each other, leading to significantly improved performance in both unsupervised and supervised tasks. Code is publicly available on GitHub.

Paper Structure

This paper contains 12 sections, 7 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. SOM Debrief
  4. ViT-SOM
  5. Experiments and Results
  6. Training
  7. Datasets.
  8. Unsupervised
  9. Supervised
  10. Unsupervised Clustering Results
  11. Classification Results
  12. Conclusion

Figures (3)

  • Figure 1: Model size (M) and task performances comparisons between ViT-SOM and baseline methods on MNIST and CIFAR-10 datasets. Model size is log scaled for interpretability.
  • Figure 2: ViT-SOM architecture.
  • Figure 3: UMAP visualization of latent space embeddings from ViT-SOM model training progression on MNIST. Projections shown at epochs 0, 25, 50, 200. Color mapping represents ground truth classes, demonstrating emergent cluster separation during training.