Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy

Albert Dominguez Mantes, Gioele La Manno, Martin Weigert

TL;DR

MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image, is introduced, demonstrating that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.

Abstract

Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.

MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy

TL;DR

MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image, is introduced, demonstrating that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.

Abstract

Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.
Paper Structure (33 sections, 4 equations, 9 figures, 8 tables)

This paper contains 33 sections, 4 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Multi-scale learning with transformers.
  4. Masked autoencoder pre-training.
  5. Rotary positional embeddings in vision.
  6. Methods
  7. Input and spatial representation
  8. Bounding box generation.
  9. MuViT encoder
  10. MuViT-MAE: Self-supervised pre-training
  11. MuViT-seg: Semantic segmentation
  12. Experiments
  13. Datasets
  14. Evaluation metrics
  15. MAE pretraining
  16. ...and 18 more sections

Figures (9)

  • Figure 1: Overview of the MuViT architecture. a)MuViT processes multiple crops of the same scene at different physical resolutions and encodes them jointly using a shared transformer with resolution-specific RoPE position encodings. As a decoder we use b) a lightweight masked autoencoding decoder that reconstructs the masked patches for each resolution level, or c) a semantic segmentation decoder that predicts the class of each pixel at the target resolution (e.g. level 1).
  • Figure 2: MuViT-MAE reconstruction results on a)Mouse and b)KPIS. Columns show masked input and reconstruction at multiple resolution levels ($l_1=1$ top, $l_2=8$ middle, $l_3=32$ bottom). The same overall masking ratio ($\rho=0.75$) is used across examples, but each column demonstrates different Dirichlet-sampled per-level masking distributions.
  • Figure 3: Semantic Segmentation results on Synthetic. Columns show: input image, ground truth, and semantic predictions for different architectures (U-Net, DeepLabV3, MuViT$_{[1]}$, MuViT$_{[1,4]}$+UNETR (naive, i.e. without cross-scale consistent positional encodings), and MuViT$_{[1,4]}$+UNETR). Top: full image ($2k\!\times\!2k\xspace$ px). Bottom: magnified regions (white boxes) showing detail comparison.
  • Figure 4: Semantic segmentation results on Mouse. Columns show: input image, ground truth, U-Net, DeepLabV3, and MuViT$_{[1,8,32]}$+UNETR semantic predictions. Top: full image ($13k\!\times\!10k\xspace$ px). Bottom: magnified regions (white boxes) showing detail comparison.
  • Figure 5: Semantic segmentation results on KPIS. Depicted in different columns are the input image, ground truth, U-Net, MuViT$_{[1]}$+UNETR, and MuViT$_{[1,8]}$+UNETR predictions. Top: full image ($52k\!\times\!56k\xspace$ px); bottom: magnified regions (white boxes) showing detail comparison.
  • ...and 4 more figures