Missing Wedge Inpainting and Joint Alignment in Electron Tomography through Implicit Neural Representations

TL;DR

This work tackles missing-wedge artifacts and misalignment in electron tomography by introducing a fully self-supervised implicit neural representation (INR) that jointly learns the 3D volume and per-image poses. The INR acts as a neural regularizer, enabling inline alignment, missing-wedge inpainting, and denoising from a single dataset without supervised training. Across simulated phantoms and experimental datasets (catalytic nanoparticles and hyperbranched nanoparticles), the INR reconstructions consistently outperform traditional methods like SIRT, particularly under aggressive missing wedges, coarse tilts, and low-dose conditions. The approach is parallelizable on multi-GPU systems and is implemented in open-source quantEM, with broad applicability to materials and atomic-resolution electron tomography, potentially enabling higher-quality reconstructions with fewer projections.

Abstract

Electron tomography is a powerful tool for understanding the morphology of materials in three dimensions, but conventional reconstruction algorithms typically suffer from missing-wedge artifacts and data misalignment imposed by experimental constraints. Recently proposed supervised machine-learning-enabled reconstruction methods to address these challenges rely on training data and are therefore difficult to generalize across materials systems. We propose a fully self-supervised implicit neural representation (INR) approach using a neural network as a regularizer. Our approach enables fast inline alignment through pose optimization, missing wedge inpainting, and denoising of low dose datasets via model regularization using only a single dataset. We apply our method to simulated and experimental data and show that it produces high-quality tomograms from diverse and information limited datasets. Our results show that INR-based self-supervised reconstructions offer high fidelity reconstructions with minimal user input and preprocessing, and can be readily applied to a wide variety of materials samples and experimental parameters.

Figures (10)

• Figure 1: Overview of the INR-based tomographic reconstruction algorithm.(a) Schematic of a tomographic experiment using STEM. An image is recorded at each tilt step as the sample is rotated around the tilt axis. (b) A neural network receives a 3-D position vector, $\vec{r} = \{x, y, z\}$ as input, and produces an intensity value, $\phi$, at that position. This way of representing the volume is called an implicit neural representation (INR). (c) The reconstruction volume is sampled by rays that terminate at individual pixels of the experimental data. Each ray also incorporates and optimizes the pose for each tilt image in the dataset. (d) Optimization loop schematic. Coordinates sampled along each ray are fed to the INR, whose outputs are integrated to predict the pixel intensity. This prediction is compared to the experimental measurement with a pixel-wise loss, which is back-propagated to update both the INR weights and the pose. (e) Schematic showing how the experimental tilt images are reshaped into a 1D array of pixels that are compared to individual pixel predictions made using the INR.
• Figure 1: HAADF, and Carbon and Titanium EDS maps of supported nanoparticles. (a) HAADF image of the sample (b) Carbon (c) and Titanium XEDS signals.
• Figure 2: Comparison between INR and SIRT for a simulated phantom. (a) Reference slice of the ground truth phantom, slices along the volume, and a 3D render. (b-d) Comparison between our method (INR) versus SIRT by varying the missing wedge size, tilt step, and electron fluence respectively through a slice perpendicular to the missing wedge direction. For each case we also compute the SSIM between each reconstructed slice to the ground truth.
• Figure 2: Comparison between smooth$_{L_1}$ (L1) and MSE (L2) loss for a simulated phantom. a(a-c) Comparison between L1 and L2 losses by varying the missing wedge size, tilt step, and electron fluence respectively through a slice perpedincular to the missing wedge direction. The SSIMs were computed using the ground truth slice.
• Figure 3: INR corrected and uncorrected pose comparison.(a) Slice through the phantom volume for varying tilt-axis error pose optimization, (b) without pose optimization. The inset shows the zoomed in circular feature in the middle of the slice. (c) SSIM for different tilt-axis errors for the corrected and uncorrected case. (d) SSIM and the number of training epochs required to converge depending on the degree of the tilt-axis error. The gray dashed line indicates where we begin optimization of the poses.