GaraMoSt: Parallel Multi-Granularity Motion and Structural Modeling for Efficient Multi-Frame Interpolation in DSA Images

TL;DR

GaraMoSt tackles the challenge of direct multi-frame interpolation for 4D DSA images by introducing a parallel, wide-network pipeline and a Multi-Granularity Motion-Structure Feature Extractor (MG-MSFE). It extracts motion and structural cues at multiple granularities across scales in parallel, uses cross-scale fusion, and decodes multiple frames in parallel via Time Mapping and a Dual-Layer Flow-Mask Estimator, followed by a Refiner that incorporates shallow structural features. The approach achieves state-of-the-art accuracy, robustness, visual fidelity, and noise suppression on DSA data while maintaining real-time inference, addressing both high-frequency and low-frequency noise that previous methods struggled with. By avoiding heavy attention maps in favor of linear context transformations and parallel processing, GaraMoSt delivers significant practical gains for real-time vascular diagnostics and interventions. The results show GaraMoSt outperforming MoSt-DSA and other natural-scene VFI methods across single- and multi-frame interpolation tasks on DSA sequences.

Abstract

The rapid and accurate direct multi-frame interpolation method for Digital Subtraction Angiography (DSA) images is crucial for reducing radiation and providing real-time assistance to physicians for precise diagnostics and treatment. DSA images contain complex vascular structures and various motions. Applying natural scene Video Frame Interpolation (VFI) methods results in motion artifacts, structural dissipation, and blurriness. Recently, MoSt-DSA has specifically addressed these issues for the first time and achieved SOTA results. However, MoSt-DSA's focus on real-time performance leads to insufficient suppression of high-frequency noise and incomplete filtering of low-frequency noise in the generated images. To address these issues within the same computational time scale, we propose GaraMoSt. Specifically, we optimize the network pipeline with a parallel design and propose a module named MG-MSFE. MG-MSFE extracts frame-relative motion and structural features at various granularities in a fully convolutional parallel manner and supports independent, flexible adjustment of context-aware granularity at different scales, thus enhancing computational efficiency and accuracy. Extensive experiments demonstrate that GaraMoSt achieves the SOTA performance in accuracy, robustness, visual effects, and noise suppression, comprehensively surpassing MoSt-DSA and other natural scene VFI methods. The code and models are available at https://github.com/ZyoungXu/GaraMoSt.

Figures (7)

• Figure 1: SSIM-Time comparison of various methods for interpolating 1 to 3 frames. Our GaraMoSt-$\mathcal{L}_1$ achieves 95.05, 94.70, 94.22 SSIM, 0.029s, 0.076s, 0.122s inference time, demonstrating SOTA accuracy, while time cost is almost the same as MoSt-DSA. Details in Table \ref{['tab:inf1_ave']},\ref{['tab:inf2_ave']},\ref{['tab:inf3_ave']}.
• Figure 2: Qualitative and quantitative comparison of GaraMoSt vs. MoSt-DSA. (a) Qualitatively, MoSt-DSA shows insufficient suppression of high and low frequency noise, whereas GaraMoSt significantly improves these issues. (b) Quantitatively, GaraMoSt enhances noise suppression, with notable reductions in the first quartile, median, third quartile, and upper adjacent of the noise distribution. Additionally, the reduction in width demonstrates a decrease in noise quantity.
• Figure 3: Illustration of our proposed parallel multi-granularity extraction and other methods of extracting motion and structural features. The approach proposed by our GaraMoSt is shown in (e). Notably, "simultaneous" should not be confused with "parallel". While (c) and (d) achieve simultaneous output of motion and structural features outside the module, the internal computation process is still highly sequential.
• Figure 4: Overall pipeline of our GaraMoSt. The encoder includes the multi-scale feature extractor (MSFE), cross-scale feature cross fusion (CSFCF), and the multi-granularity motion-structure feature extractor (MG-MSFE) for extracting general multi-granularity motion and structural features in parallel. The decoder consists of the time mapping (TM), dual-layer flow-mask estimator (DL-FME), and Refiner modules, tailored to parallel decode and refine general multi-granularity features to generate ${I_t}$. Notably, ${t}$ can be any value between $[0,1]$, and multi ${t}$ can be specified at once forward calculation, thus enabling direct multi-frame interpolation in both training and inference.
• Figure 5: Multi-Granularity MoSt-Attention for parallel extracting of multi-granularity motion and structural features. Different scales correspond to different values of $s$, with $s$ being either $1$ or $2$. In the left image pyramid, the yellow and blue blocks represent the cross-scale fusion features of ${I_0}$ and ${I_1}$ respectively. Enhanced structural features are used in MG-MSFE for subsequent calculations to derive the final structural features, see Figure \ref{['fig:overall_pipeline']} for details.