Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Pretraining-Induced Attention Asymmetry in 2D+ Electron Microscopy Image Segmentation

Zsófia Molnár, Gergely Szabó, András Horváth

TL;DR

A targeted modification of pretraining weights based on uniform channel initialization was proposed, which restores symmetric feature attribution while preserving the benefits of pretraining and confirms a substantial reduction in attribution bias without compromising or even improving segmentation accuracy.

Abstract

Vision models pretrained on large-scale RGB natural image datasets are widely reused for electron microscopy image segmentation. In electron microscopy, volumetric data are acquired as serial sections and processed as stacks of adjacent grayscale slices, where neighboring slices provide symmetric contextual information for identifying features on the central slice. The common strategy maps such stacks to pseudo-RGB inputs to enable transfer learning from pretrained models. However, this mapping imposes channel-specific semantics inherited from natural images, even though electron microscopy slices are homogeneous in the modality and symmetric in their predictive roles. As a result, pretrained models may encode inductive biases that are misaligned with the inherent symmetry of volumetric electron microscopy data. In this work, it is demonstrated that RGB-pretrained models systematically assign unequal importance to individual input slices when applied to stacked electron microscopy data, despite the absence of any intrinsic channel ordering. Using saliency-based attribution analysis across multiple architectures, a consistent channel-level asymmetry was observed that persists after fine-tuning and affects model interpretability, even when segmentation performance is unchanged. To address this issue, a targeted modification of pretraining weights based on uniform channel initialization was proposed, which restores symmetric feature attribution while preserving the benefits of pretraining. Experiments on the SNEMI, Lucchi and GF-PA66 datasets confirm a substantial reduction in attribution bias without compromising or even improving segmentation accuracy.

Mitigating Pretraining-Induced Attention Asymmetry in 2D+ Electron Microscopy Image Segmentation

TL;DR

A targeted modification of pretraining weights based on uniform channel initialization was proposed, which restores symmetric feature attribution while preserving the benefits of pretraining and confirms a substantial reduction in attribution bias without compromising or even improving segmentation accuracy.

Abstract

Vision models pretrained on large-scale RGB natural image datasets are widely reused for electron microscopy image segmentation. In electron microscopy, volumetric data are acquired as serial sections and processed as stacks of adjacent grayscale slices, where neighboring slices provide symmetric contextual information for identifying features on the central slice. The common strategy maps such stacks to pseudo-RGB inputs to enable transfer learning from pretrained models. However, this mapping imposes channel-specific semantics inherited from natural images, even though electron microscopy slices are homogeneous in the modality and symmetric in their predictive roles. As a result, pretrained models may encode inductive biases that are misaligned with the inherent symmetry of volumetric electron microscopy data. In this work, it is demonstrated that RGB-pretrained models systematically assign unequal importance to individual input slices when applied to stacked electron microscopy data, despite the absence of any intrinsic channel ordering. Using saliency-based attribution analysis across multiple architectures, a consistent channel-level asymmetry was observed that persists after fine-tuning and affects model interpretability, even when segmentation performance is unchanged. To address this issue, a targeted modification of pretraining weights based on uniform channel initialization was proposed, which restores symmetric feature attribution while preserving the benefits of pretraining. Experiments on the SNEMI, Lucchi and GF-PA66 datasets confirm a substantial reduction in attribution bias without compromising or even improving segmentation accuracy.

Paper Structure

This paper contains 21 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Pretrained Models and Transfer Learning
  4. Challenges and Benefits of 2D+ Representations in Imaging
  5. Saliency Asymmetry and Attention Mechanisms
  6. Datasets
  7. SNEMI dataset
  8. Lucchi dataset
  9. GF-PA66 3D XCT dataset
  10. Experimental Design
  11. Model Architectures and Training Setup
  12. Saliency Pattern Evaluation Setup
  13. Channel Attention Difference Measures
  14. Results
  15. Segmentation Performances
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Visualization of a randomly selected SNEMI image alongside various saliency map types used in this study. The saliency map visualizations underwent histogram equalization to ensure a comparable scale across different methods.
  • Figure 2: Example images from the three different datasets used in the study. The top row contains raw input images, while the bottom row shows their corresponding segmentation masks.
  • Figure 3: Example prediction result for the SNEMI dataset. The predicted mask is thresholded at the confidence value of 0.85.
  • Figure 4: SNEMI dataset example of averaged "Foreground" saliency map calculation using ImageNet and "Uniform-Green" pretrained models, showing highly variable channel attention in ImageNet, while showing almost no variance in "Uniform-Green".