SAM-EM: Real-Time Segmentation for Automated Liquid Phase Transmission Electron Microscopy

TL;DR

SAM-EM addresses the challenge of real-time segmentation and tracking in noisy LPTEM videos by domain-adapting the Segment Anything Model 2 (SAM 2) through full-model fine-tuning on a synthetic LPTEM dataset. The approach unifies segmentation, tracking, and trajectory analysis (including mean-squared displacement and displacement distributions) in a single pipeline, with robustness under low SNR encountered in thicker liquid layers. It demonstrates superior mask fidelity and temporal identity stability over zero-shot SAM-2 and a U-Net baseline, enabling reliable quantitative nanoscale dynamics extraction and potential closed-loop LPTEM workflows. By releasing code and datasets, SAM-EM aims to accelerate data-driven materials discovery and design through real-time, interpretable analysis of in situ electron microscopy data.

Abstract

The absence of robust segmentation frameworks for noisy liquid phase transmission electron microscopy (LPTEM) videos prevents reliable extraction of particle trajectories, creating a major barrier to quantitative analysis and to connecting observed dynamics with materials characterization and design. To address this challenge, we present Segment Anything Model for Electron Microscopy (SAM-EM), a domain-adapted foundation model that unifies segmentation, tracking, and statistical analysis for LPTEM data. Built on Segment Anything Model 2 (SAM~2), SAM-EM is derived through full-model fine-tuning on 46,600 curated LPTEM synthetic video frames, substantially improving mask quality and temporal identity stability compared to zero-shot SAM~2 and existing baselines. Beyond segmentation, SAM-EM integrates particle tracking with statistical tools, including mean-squared displacement and particle displacement distribution analysis, providing an end-to-end framework for extracting and interpreting nanoscale dynamics. Crucially, full fine-tuning allows SAM-EM to remain robust under low signal-to-noise conditions, such as those caused by increased liquid sample thickness in LPTEM experiments. By establishing a reliable analysis pipeline, SAM-EM transforms LPTEM into a quantitative single-particle tracking platform and accelerates its integration into data-driven materials discovery and design. Project page: \href{https://github.com/JamaliLab/SAM-EM}{github.com/JamaliLab/SAM-EM}.

Figures (9)

• Figure 1: SAM-EM framework for particle segmentation and tracking in LPTEM.a, The segmentation module segments an LPTEM video with multiple nanoparticles, each prompted once on the first frame. b, Deployment on any Python or Docker system with a graphical interface that enables particle selection via bounding-box prompts and statistical test selection. c, Extracted trajectories undergo quantitative analyses, including spatiotemporal paths (color-coded over 0-45 seconds), mean squared displacements, displacement distributions, and velocity autocorrelations. Scale bar, 250 nm.
• Figure 2: Performance accuracy of SAM-EM in segmentation on simulated datasets. Comparison of fine-tuned SAM-EM model and baseline (zero-shot SAM 2) on a simulated test dataset with 300 frames with two moving nanoparticles and with minimal particle interaction.
• Figure 3: Example simulated LPTEM frames at different liquid thicknesses, illustrating the decrease in image SNR as thickness (and electron scattering) increase.
• Figure 4: Performance accuracy of SAM-EM in segmentation on a simulated dataset with resolution of 512$\times$512 resolution. Comparison of fine-tuned SAM-EM model and baseline (zero-shot SAM 2) on a simulated test dataset with 275 frames with one moving nanoparticle.
• Figure 5: Performance accuracy of SAM-EM in segmentation on simulated datasets with resolution of 1024$\times$1024. Comparison of fine-tuned SAM-EM model and baseline (zero-shot SAM 2) on a simulated test dataset with 600 frames with four moving nanoparticles and with minimal particle interaction.