Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAM4EM: Efficient memory-based two stage prompt-free segment anything model adapter for complex 3D neuroscience electron microscopy stacks

Uzair Shah, Marco Agus, Daniya Boges, Vanessa Chiappini, Mahmood Alzubaidi, Jens Schneider, Markus Hadwiger, Pierre J. Magistretti, Mowafa Househ, Corrado Calı

TL;DR

SAM4EM addresses the challenge of 3D EM segmentation with limited annotated data by extending SAM with a prompt-free, memory-informed two-stage adapter. It combines a LoRA-adapted image encoder, a multi-scale feature enhancer, an efficient memory encoder, and bi-directional self-prompting to enforce temporal consistency across slices, followed by a two-stage hierarchical mask decoding. The approach yields state-of-the-art performance on mitochondria, glia, and synapse segmentation in EM stacks and introduces a new dataset for astrocytic processes and synapses. This work enables accurate, scalable segmentation of complex neural structures with reduced data and compute, offering practical impact for nanoscale neuroscience analysis and EM-based pipelines.

Abstract

We present SAM4EM, a novel approach for 3D segmentation of complex neural structures in electron microscopy (EM) data by leveraging the Segment Anything Model (SAM) alongside advanced fine-tuning strategies. Our contributions include the development of a prompt-free adapter for SAM using two stage mask decoding to automatically generate prompt embeddings, a dual-stage fine-tuning method based on Low-Rank Adaptation (LoRA) for enhancing segmentation with limited annotated data, and a 3D memory attention mechanism to ensure segmentation consistency across 3D stacks. We further release a unique benchmark dataset for the segmentation of astrocytic processes and synapses. We evaluated our method on challenging neuroscience segmentation benchmarks, specifically targeting mitochondria, glia, and synapses, with significant accuracy improvements over state-of-the-art (SOTA) methods, including recent SAM-based adapters developed for the medical domain and other vision transformer-based approaches. Experimental results indicate that our approach outperforms existing solutions in the segmentation of complex processes like glia and post-synaptic densities. Our code and models are available at https://github.com/Uzshah/SAM4EM.

SAM4EM: Efficient memory-based two stage prompt-free segment anything model adapter for complex 3D neuroscience electron microscopy stacks

TL;DR

SAM4EM addresses the challenge of 3D EM segmentation with limited annotated data by extending SAM with a prompt-free, memory-informed two-stage adapter. It combines a LoRA-adapted image encoder, a multi-scale feature enhancer, an efficient memory encoder, and bi-directional self-prompting to enforce temporal consistency across slices, followed by a two-stage hierarchical mask decoding. The approach yields state-of-the-art performance on mitochondria, glia, and synapse segmentation in EM stacks and introduces a new dataset for astrocytic processes and synapses. This work enables accurate, scalable segmentation of complex neural structures with reduced data and compute, offering practical impact for nanoscale neuroscience analysis and EM-based pipelines.

Abstract

We present SAM4EM, a novel approach for 3D segmentation of complex neural structures in electron microscopy (EM) data by leveraging the Segment Anything Model (SAM) alongside advanced fine-tuning strategies. Our contributions include the development of a prompt-free adapter for SAM using two stage mask decoding to automatically generate prompt embeddings, a dual-stage fine-tuning method based on Low-Rank Adaptation (LoRA) for enhancing segmentation with limited annotated data, and a 3D memory attention mechanism to ensure segmentation consistency across 3D stacks. We further release a unique benchmark dataset for the segmentation of astrocytic processes and synapses. We evaluated our method on challenging neuroscience segmentation benchmarks, specifically targeting mitochondria, glia, and synapses, with significant accuracy improvements over state-of-the-art (SOTA) methods, including recent SAM-based adapters developed for the medical domain and other vision transformer-based approaches. Experimental results indicate that our approach outperforms existing solutions in the segmentation of complex processes like glia and post-synaptic densities. Our code and models are available at https://github.com/Uzshah/SAM4EM.
Paper Structure (13 sections, 3 equations, 5 figures, 2 tables)

This paper contains 13 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. SAM4EM Architecture
  5. Overview
  6. LoRA-adapted Image Encoder
  7. Feature Enhancer Block
  8. Efficient Memory Encoder
  9. Bi-directional Self-Prompting
  10. Experimental Setup
  11. Implementation details
  12. Results
  13. Conclusion

Figures (5)

  • Figure 1: Current SOTA segmentation methods for 3D EM are effective for well-defined cellular structures like mitochondria Luo:2024:frag, while they struggle when segmenting complex processes with fractal structures like glia cells. We propose an efficient adapter for the Segment Anything Model which overcomes this limitation. Left: EM image. Center: glia annotation. Right: full glia 3D reconstruction.
  • Figure 2: Overview of the proposed SAM4EM architecture. Our model extends SAM with three key components: (a) feature enhancer block (Sec. \ref{['sec:enhancer']}), (b) memory-based encoder (Sec. \ref{['sec:memory']}), and (c) bi-directional prompt mechanism (Sec. \ref{['sec:prompt']}), integrated with residual connections for accurate medical image segmentation.
  • Figure 3: Architecture of the proposed feature enhancer block. Multi-resolution features ($1/4$, $1/8$, $1/16$) are processed through parallel pathways with up-sampling, smoothing convolutions, and residual connections. Features are fused via concatenation and refined through channel reduction and dual convolution blocks for precise boundary delineation.
  • Figure 4: Architecture of the efficient memory encoder. Image and mask features are processed through parallel streams with feature projection and mask downsampling, then integrated with a memory buffer (M=8) via attention mechanism. Moving average updates and skip connections maintain temporal consistency for feature conditioning.
  • Figure 5: Qualitative comparison of segmentation results across different datasets and methods. From left to right: input electron microscopy images, ground truth masks, and segmentation results from our proposed method, H-SAM, SAMed, and UN-SAM. The comparison spans four different datasets: Lucchi (mitochondria), Glia (glial cells), Mito (mitochondria), and Synapse (synaptic junctions). Our method demonstrates superior boundary delineation and structural preservation across varied anatomical structures, particularly evident in the complex morphologies of glial cells and the fine details of synaptic junctions.