Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images

TL;DR

This work addresses the challenge of interpolating 4D medical images without intermediate frames by introducing UVI-Net, a data-efficient, unsupervised framework. It uses a two-stage, flow-based approach with virtual intermediate samples and a cycle-consistency constraint to reconstruct authentic frames, achieving state-of-the-art performance among unsupervised and supervised baselines. The method demonstrates robustness to limited data, including successful interpolation when trained on a single data pair, and gains can be further amplified via instance-specific optimization at test time. The approach also enables effective 3D data augmentation for downstream segmentation tasks, highlighting practical impact for clinical imaging where data are scarce and expensive to acquire.

Abstract

4D medical images, which represent 3D images with temporal information, are crucial in clinical practice for capturing dynamic changes and monitoring long-term disease progression. However, acquiring 4D medical images poses challenges due to factors such as radiation exposure and imaging duration, necessitating a balance between achieving high temporal resolution and minimizing adverse effects. Given these circumstances, not only is data acquisition challenging, but increasing the frame rate for each dataset also proves difficult. To address this challenge, this paper proposes a simple yet effective Unsupervised Volumetric Interpolation framework, UVI-Net. This framework facilitates temporal interpolation without the need for any intermediate frames, distinguishing it from the majority of other existing unsupervised methods. Experiments on benchmark datasets demonstrate significant improvements across diverse evaluation metrics compared to unsupervised and supervised baselines. Remarkably, our approach achieves this superior performance even when trained with a dataset as small as one, highlighting its exceptional robustness and efficiency in scenarios with sparse supervision. This positions UVI-Net as a compelling alternative for 4D medical imaging, particularly in settings where data availability is limited. The source code is available at https://github.com/jungeun122333/UVI-Net.

Figures (10)

• Figure 1: An overview of time-domain cycle consistency constraint. This image illustrates the process of generating $\hat{I}_0^{cyc}$. (1) $I_0$ and $I_1$ are given two input frames, with $I_1$ ommited for sake of readability. (2) We first generate virtual intermediate frames, and (3) subsequently generate back the frames with multi-resolution features (denoted as blue cubics). (4) The resulting reconstructed images $\hat{I}_0^{cyc}$ must match the original input frame, $I_0$.
• Figure 2: Schematic overview of our entire inference process. Starting with two input frames, $I_0$ and $I_1$, we input the frames into the flow calculation model to obtain the approximated flow fields $\phi_{0 \rightarrow t}$ and $\phi_{1 \rightarrow t}$. We then warp the two frames using the obtained flow field, and similarly warp the multi-scale voxelwise features. Finally, we refine the distance-inversely weighted added image considering the information from multi-scale features, resulting in the final interpolated frame $\hat{I}_t$.
• Figure 3: Architecture of the feature extractor module based on 3D Convolutional Neural Network (CNN). $h$, $w$, and $d$ are the input image's height, width, and depth, respectively.
• Figure 4: Performance trends based on the size of the training datasets. The dashed line represents a supervised setting. As depicted in this figure, we observe that the performance gap between our model and the baselines increases regardless of whether the setting is supervised or not, and irrespective of the dataset type. This demonstrates our model's robustness, particularly in addressing data scarcity issues common in the medical domain.
• Figure 5: Visualization examples from 4D cardiac and lung datasets. The model marked with an '*' is trained exclusively on the test set, while models marked with '(SL)' are trained using supervised learning. Our method generates intermediate frames that are not only visually appealing but also precise, successfully retaining fine details and maintaining the structural integrity of the original images.