Deep learning-based conditional inpainting for restoration of artifact-affected 4D CT images

Abstract

4D CT imaging is an essential component of radiotherapy of thoracic/abdominal tumors. 4D CT images are, however, often affected by artifacts that compromise treatment planning quality. In this work, deep learning (DL)-based conditional inpainting is proposed to restore anatomically correct image information of artifact-affected areas. The restoration approach consists of a two-stage process: DL-based detection of common interpolation (INT) and double structure (DS) artifacts, followed by conditional inpainting applied to the artifact areas. In this context, conditional refers to a guidance of the inpainting process by patient-specific image data to ensure anatomically reliable results. The study is based on 65 in-house 4D CT images of lung cancer patients (48 with only slight artifacts, 17 with pronounced artifacts) and two publicly available 4D CT data sets that serve as independent external test sets. Automated artifact detection revealed a ROC-AUC of 0.99 for INT and of 0.97 for DS artifacts (in-house data). The proposed inpainting method decreased the average root mean squared error (RMSE) by 52%(INT) and 59% (DS) for the in-house data. For the external test data sets, the RMSE improvement is similar (50% and 59 %, respectively). Applied to 4D CT data with pronounced artifacts (not part of the training set), 72% of the detectable artifacts were removed. The results highlight the potential of DL-based inpainting for restoration of artifact-affected 4D CT data. Compared to recent 4D CT inpainting and restoration approaches, the proposed methodology illustrates the advantages of exploiting patient-specific prior image information.

Figures (6)

• Figure 1: Left: Visualization of double structure (DS) artifacts (subfigure a, artifact area highlighted by red box) and interpolation (INT) artifacts (d, artifact areas highlighted by blue boxes) in 4D CT image data. Right: Illustration of direct 2D conditional inpainting on 4D CT data affected by DS artifacts. Subfigure b shows an artifact (highlighted in red) for a patient with a small breathing amplitude and subfigure e the same coronal slice of the 3D CT volume after 2D inpainting. In c, a DS artifact (highlighted in red) is depicted for a patient with large breathing amplitudes; the corresponding coronal slice after 2D inpainting is shown in f. As visible, a 2D-based approach is able to properly inpaint smaller artifacts; however, it fails for more pronounced patient motion and artifacts. INT artifacts have not yet been addressed by inpainting approaches.
• Figure 2: Flowchart of our proposed artifact detection and inpainting approach. Note that the information flow and the network structure is similar for INT and DS artifacts. However, different conditional images $\mathbf{I}_\mathrm{C}$ are used and artifact type-specific networks are trained. For both artifact types, prior to inpainting, an artifact type-specific mask $\mathbf{M}_\mathrm{A}$ is predicted for each 3D phase CT by application of an artifact detection network (normalization: batch norm; activation [block 0(a) and (b)]: ReLU; pooling: max pool, activation [block 0(c)]: sigmoid). The detection is performed slice-wise for all coronal CT slices $\mathbf{I}_{A,y}$ overlapping with the lung bounding box of the 3D CT image. The results are combined as detailed in the main text; the output is a 3-dimensional binary mask $\mathbf{M}_\mathrm{A}$ (1: artifact, 0: else). Subsequently, the conditional image $\mathbf{I}_\mathrm{C}$, artifact-affected image $\mathbf{I}_\mathrm{A}$ and the predicted $\mathbf{M}_\mathrm{A}$ are fed into block 1, the spatial transformer block. In sub-block (a), a vector field $\varphi$ that aims at maximizing the similarity of $\mathbf{I}_\mathrm{C}$ and $\mathbf{I}_\mathrm{A}\odot(\textbf{1}-\mathbf{M}_\mathrm{A})$ is estimated by demons-based DIR ($\mathcal{P}$: image pyramid; for parameter details see main text). An end-to-end trainable vector field correction network (activation: Mish, normalization: instance norm), as illustrated in sub-block (b), improves the resulting transformation by predicting a vector field correction $\Delta\varphi_\mathrm{corr}$, resulting in a final corrected transformation $\varphi_\mathrm{corr}$. Finally, artifact inpainting is performed in block 2 using a residual dense net (RDN) with inputs being the transformed conditional image ($\mathbf{I}_\mathrm{C}\circ\varphi$), $\mathbf{I}_\mathrm{A}$ and corresponding $\mathbf{M}_\mathrm{A}$. Here, the first sub-block (feature extend block, followed by two residual dense blocks with standard convolutions and a feature fuse block) extracts feature maps based on the masked artifact-affected image and the warped conditional image, which are employed in the second sub-block (similar structure, but using partial convolutions) to inpaint the artifact region defined by $\mathbf{M}_\mathrm{A}$.
• Figure 3: a) INT and DS artifact size distributions for a subset of the in-house data set. b) Illustration of simulated INT (highlighted by blue box) and c) DS (red box) artifacts.
• Figure 4: Inpainting performance of the Full model configurations with Conv/PConv layers for simulated artifacts and the in-house validation data, quantified by $\Delta$RMSE (left) and NCC (right) values.
• Figure 5: DS/INT inpainting results for the configurations Full model, Inpaint [$\mathbf{I}_\mathrm{C}\circ\varphi$] and Inpaint [$\mathbf{I}_\mathrm{C}$] (DS: c--e/INT: f--h). The initial artifact images $\mathbf{I}_\mathrm{A}$ (artifact area marked in red/blue for DS/INT artifacts), conditional images $\mathbf{I}_\mathrm{C}$, and ground truth (GT), i.e., artifact-free images $\mathbf{I}$, are shown in blocks a and b for DS and INT artifacts, respectively. For each inpainting result, the squared intensity difference with respect to the GT image is shown (color-coded plots, areas outside the overlaid artifact masks are not modified, i.e., $\Delta \text{HU}^2 = 0$), and corresponding line profiles (summed differences in LR direction) are plotted in comparison to the artifact-affected image in the bottom row.