Adapting SAM for Volumetric X-Ray Data-sets of Arbitrary Sizes

TL;DR

The research indicates the potential of combining SAM with FFN for volumetric instance segmentation tasks, particularly in NDT scenarios and segmenting large entities and objects.

Abstract

Objective: We propose a new approach for volumetric instance segmentation in X-ray Computed Tomography (CT) data for Non-Destructive Testing (NDT) by combining the Segment Anything Model (SAM) with tile-based Flood Filling Networks (FFN). Our work evaluates the performance of SAM on volumetric NDT data-sets and demonstrates its effectiveness to segment instances in challenging imaging scenarios. Methods: We implemented and evaluated techniques to extend the image-based SAM algorithm fo the use with volumetric data-sets, enabling the segmentation of three-dimensional objects using FFN's spatially adaptability. The tile-based approach for SAM leverages FFN's capabilities to segment objects of any size. We also explore the use of dense prompts to guide SAM in combining segmented tiles for improved segmentation accuracy. Results: Our research indicates the potential of combining SAM with FFN for volumetric instance segmentation tasks, particularly in NDT scenarios and segmenting large entities and objects. Conclusion: While acknowledging remaining limitations, our study provides insights and establishes a foundation for advancements in instance segmentation in NDT scenarios.

Figures (20)

• Figure 1: Rendered example of instance segmentation (Figure \ref{['fig:introduction-task-segmentation']}) of a sub-volume of size $512\times512\times512$ voxels (Figure \ref{['fig:introduction-task-subvolume']}) from the XXL-CT Me 163 data-set with a data resolution of $10000.0\times10000.0\times8000.0$ voxels (Figure \ref{['fig:introduction-task-reko']}).
• Figure 2: Photographs, exemplary CT slices, and reference segmentation of the Me 163 (Figures \ref{['fig:methods-datasets-me163-photo']}, \ref{['fig:methods-datasets-me163-reco']}, \ref{['fig:methods-datasets-me163-reference']}), marbles (Figures \ref{['fig:methods-datasets-marbles-photo']}, \ref{['fig:methods-datasets-marbles-reco']}, \ref{['fig:methods-datasets-marbles-reference']}), and corn (Figures \ref{['fig:methods-datasets-corn-photo']}, \ref{['fig:methods-datasets-corn-reco']}, \ref{['fig:methods-datasets-corn-reference']}) data-sets respectively.
• Figure 3: Zero-padding preparation steps were performed on the input and reference slices of the different data-sets to create slices of size $1024\times1024$ pixels centred around each possible seed point. The white border regions in the available input and reference slices were filled with constant values of zero.
• Figure 4: Processing of an example foreground slice used for fine tuning SAM. Consisting of reconstruction slice (Figure \ref{['fig:methods-fineTune-dataset-foreground-input']}), reference slice (Figure \ref{['fig:methods-fineTune-dataset-foreground-reference']}), one-hot encoded (Figure \ref{['fig:methods-fineTune-dataset-foreground-oneHot']}), and connected component training target (Figure \ref{['fig:methods-fineTune-dataset-forground-ConnectedComponentTarget']}). The green cross marks the centre of the slice.
• Figure 5: Processing of an example background slice used for fine tuning SAM. The green cross marks the centre of the slice, which is located in the background of the reconstruction. The green border around the reconstruction slice in Figure \ref{['fig:methods-fineTune-dataset-background-input']} depicts the original volume size, which then was enframed with an constant value border. The other sub-figures show the tested possibilities for target slices for the fine-tuning: ForegroundOnly (Figure \ref{['fig:methods-fineTune-dataset-background-noBackground']}), ConstantValueBackground (Figure \ref{['fig:methods-fineTune-dataset-background-constantValueBackground']}), and ConnectedComponentBackground (Figure\ref{['fig:methods-fineTune-dataset-background-connectedComponentBackground']}).