Triggering hallucinations in model-based MRI reconstruction via adversarial perturbations

TL;DR

The results show that these models are highly susceptible to small perturbations and can be easily coaxed into producing hallucinations, which may partially explain why hallucinations occur in the first place and suggest that a carefully constructed adversarial training routine may reduce their prevalence.

Abstract

Generative models are increasingly used to improve the quality of medical imaging, such as reconstruction of magnetic resonance images and computed tomography. However, it is well-known that such models are susceptible to hallucinations: they may insert features into the reconstructed image which are not actually present in the original image. In a medical setting, such hallucinations may endanger patient health as they can lead to incorrect diagnoses. In this work, we aim to quantify the extent to which state-of-the-art generative models suffer from hallucinations in the context of magnetic resonance image reconstruction. Specifically, we craft adversarial perturbations resembling random noise for the unprocessed input images which induce hallucinations when reconstructed using a generative model. We perform this evaluation on the brain and knee images from the fastMRI data set using UNet and end-to-end VarNet architectures to reconstruct the images. Our results show that these models are highly susceptible to small perturbations and can be easily coaxed into producing hallucinations. This fragility may partially explain why hallucinations occur in the first place and suggests that a carefully constructed adversarial training routine may reduce their prevalence. Moreover, these hallucinations cannot be reliably detected using traditional image quality metrics. Novel approaches will therefore need to be developed to detect when hallucinations have occurred.

Figures (8)

• Figure 1: Comparison of classical FISTA and model-based FISTA-Net reconstruction of brain MR images. These images were taken from aromal2024fista.
• Figure 2: Examples of hallucinations introduced by model-based reconstructions of MR images. (a) In this image from muckley2020state, the model-based reconstruction introduces an additional sulcus in the brain which was not present in the original. (b) In this image taken from cheng2020addressing, the model-based reconstruction removes evidence of a meniscal tear that is present in the original data.
• Figure 3: Distributions of the metric values for the UNet model.
• Figure 4: Distributions of the metric values for the E2E-VarNet model.
• Figure 5: Distributions of the metric values for the UNet model using total variation reconstructions.