Quantification and Classification of Carbon Nanotubes in Electron Micrographs using Vision Foundation Models

TL;DR

This work presents a two-stage framework for automated CNT quantification and morphology classification in electron micrographs by integrating vision foundation models. A SAM-based interactive segmentation tool provides high-quality particle masks, which are then used to guide a DINOv2-based classifier through mask-aware, hypercolumn feature extraction and pooling. Across 1,800 TEM images, the approach achieves a held-out test accuracy of $95.5\%$, with an overall dataset accuracy of $98.5\%$, significantly exceeding prior benchmarks while using substantially less labeled data. The method enables robust analysis of mixed CNT samples within a single field of view and demonstrates the potential of zero-shot segmentation combined with self-supervised representations to accelerate nanomaterial characterization in occupational health and materials science. This work also outlines pathways to domain-specific foundation models and multi-modal extensions to further improve accuracy and applicability in EM-based nanomaterial analysis.

Abstract

Accurate characterization of carbon nanotube morphologies in electron microscopy images is vital for exposure assessment and toxicological studies, yet current workflows rely on slow, subjective manual segmentation. This work presents a unified framework leveraging vision foundation models to automate the quantification and classification of CNTs in electron microscopy images. First, we introduce an interactive quantification tool built on the Segment Anything Model (SAM) that segments particles with near-perfect accuracy using minimal user input. Second, we propose a novel classification pipeline that utilizes these segmentation masks to spatially constrain a DINOv2 vision transformer, extracting features exclusively from particle regions while suppressing background noise. Evaluated on a dataset of 1,800 TEM images, this architecture achieves 95.5% accuracy in distinguishing between four different CNT morphologies, significantly outperforming the current baseline despite using a fraction of the training data. Crucially, this instance-level processing allows the framework to resolve mixed samples, correctly classifying distinct particle types co-existing within a single field of view. These results demonstrate that integrating zero-shot segmentation with self-supervised feature learning enables high-throughput, reproducible nanomaterial analysis, transforming a labor-intensive bottleneck into a scalable, data-driven process.

Figures (9)

• Figure 1: Representative microscopy images from the segmentation validation dataset. (a--h) SEM images of hydrophilic and hydrophobic CNT samples on polycarbonate filters displaying diverse aggregation states and morphologies including isolated particles, small clusters, and large agglomerates. (i--p) TEM images showing individual CNT structures with varying morphologies including single fibers, bundles and clusters.
• Figure 2: Classification dataset (NIOSH): Representative TEM images organized by morphology class. (a) Fiber: elongated structures with high aspect ratios, (b) Cluster: dense, non-linear agglomerates with entangled CNTs, (c) Matrix: particles embedded within web-like matrices, and (d) Matrix Surface: CNTs extruding from particle surfaces.
• Figure 3: Dual-pathway architecture for CNT morphology analysis integrating SAM and DINOv2. The Segment Anything Model (SAM ViT-B/16) generates binary segmentation masks from input SEM images for downstream quantification tasks (counting, sizing). DINOv2 (ViT-B/14) extracts multi-scale feature representations across 12 transformer blocks, visualized as stacked activation maps. The segmentation mask is applied to suppress background regions, creating masked feature maps (red borders) that focus on CNT morphology. Features from multiple layers are aggregated through hypercolumn pooling into a 3840-dimensional vector, then classified via a fully-connected network to predict four morphology classes: Fiber, Cluster, Matrix, or MatrixSurface. The convergent arrows illustrate multi-scale feature pooling from different network depths. Note: Input image dimensions differ between pathways due to distinct patch size requirements (SAM: 16$\times$16, DINOv2: 14$\times$14).
• Figure 4: Comparison of original microscopy images, ground truth segmentation masks, and overlay visualizations for different sample types. For each row, the left panel shows the original microscopy image (SEM for rows 1-2, TEM for rows 3-4), the center panel presents the binary ground truth mask, and the right panel displays the ground truth overlay in green on the original image, highlighting the segmented regions of interest.
• Figure 5: Comparative classification performance of the best-performing model. (a) Normalized confusion matrix evaluated on the entire dataset, achieving 98.5% overall accuracy. The strong diagonal demonstrates robust feature learning, with Matrix and MatrixSurface achieving near-perfect accuracy. Minor confusions occur primarily between morphologically adjacent classes: Fiber-to-Cluster (2%) and Cluster-to-MatrixSurface (2%). (b) Normalized confusion matrix evaluated on the held-out test set, achieving 95.5% overall accuracy. While Cluster (98%) and Matrix (98%) maintain strong performance, Fiber accuracy decreases to 93% with 5% misclassified as Clusters and 2% as Matrix. MatrixSurface achieves 96% accuracy with 4% confused with Matrix. These test set errors reflect genuine morphological ambiguity in distinguishing overlapping fibers from loose bundles and in differentiating dense matrix regions from matrix-surface transitions.