Pruning Sparse Tensor Neural Networks Enables Deep Learning for 3D Ultrasound Localization Microscopy

TL;DR

This work addresses the memory bottleneck that has limited applying deep learning to 3D Ultrasound Localization Microscopy (ULM). It proposes Sparse Tensor Neural Networks and a dense-to-sparse input filtering strategy to dramatically reduce memory usage while enabling 3D ULM, and it evaluates 2D and 3D performance against conventional ULM and dense DL baselines. Across 2D, sparsity yields about a twofold memory reduction with a modest drop in Dice, while in 3D it reduces memory by roughly two orders of magnitude and substantially outperforms conventional ULM at high microbubble concentrations, suggesting gains in acquisition time and throughput. The study also explores/demonstrates dense-to-sparse strategies and architecture tweaks, concluding that sparsity is the primary driver of memory efficiency with the 3D extension showing the most pronounced benefits, albeit with some trade-offs in performance in certain configurations.

Abstract

Ultrasound Localization Microscopy (ULM) is a non-invasive technique that allows for the imaging of micro-vessels in vivo, at depth and with a resolution on the order of ten microns. ULM is based on the sub-resolution localization of individual microbubbles injected in the bloodstream. Mapping the whole angioarchitecture requires the accumulation of microbubbles trajectories from thousands of frames, typically acquired over a few minutes. ULM acquisition times can be reduced by increasing the microbubble concentration, but requires more advanced algorithms to detect them individually. Several deep learning approaches have been proposed for this task, but they remain limited to 2D imaging, in part due to the associated large memory requirements. Herein, we propose to use sparse tensor neural networks to reduce memory usage in 2D and to improve the scaling of the memory requirement for the extension of deep learning architecture to 3D. We study several approaches to efficiently convert ultrasound data into a sparse format and study the impact of the associated loss of information. When applied in 2D, the sparse formulation reduces the memory requirements by a factor 2 at the cost of a small reduction of performance when compared against dense networks. In 3D, the proposed approach reduces memory requirements by two order of magnitude while largely outperforming conventional ULM in high concentration settings. We show that Sparse Tensor Neural Networks in 3D ULM allow for the same benefits as dense deep learning based method in 2D ULM i.e. the use of higher concentration in silico and reduced acquisition time.

Figures (9)

• Figure 1: The left column represents the filtered microbubble signal (i.e, the input of the network), the center column represents the corresponding microbubble tracks (i.e., the desired output of the network) and the right column represent the final result after summation of all the predictions from a dataset (i.e., the vascular structure imaged)
• Figure 2: The top row shows a dense representation of a trajectory in Deep-stULM as well as the intermediate map dimension. The bottom row illustrates how sparse formulations could reduce the memory cost: the green pixels represent the pixels of interest at each resolution, and the gray pixels represent the pixels removed through pruning based on intermediate prediction
• Figure 3: Comparison of performance under increasing concentration for conventional ULM (center left column), Deep-stULM dense formulation (center right column) and its sparse formulation (right column). Ground truth is given for comparison (left column). The scale bar is 98µ and corresponds to the wavelength of the simulated pulse. Concentration increases from 1 (top row), 5, 10, and 20 (bottom row) microbubbles per field of view.
• Figure 4: Evolution of the angiogram reconstruction performance on vascular network datasets with increasing concentration
• Figure 5: Evolution of the trajectory detection on random trajectory datasets with increasing concentration