Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tackling Structural Hallucination in Image Translation with Local Diffusion

Seunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry F. J. Tregidgo, Asher Mullokandov, Philip Teare, Daniel C. Alexander

TL;DR

This paper addresses structural hallucination in conditional diffusion models when the input contains out-of-distribution (OOD) regions, a problem that risks misdiagnosis in medical imaging and related tasks. It introduces a training-free Local Diffusion framework that first estimates OOD regions, then runs parallel branching denoising for IND and OOD areas and fuses the results to produce coherent outputs, optionally guided by an auxiliary classifier to balance phases. The authors demonstrate significant reductions in hallucinations and improved downstream task performance (e.g., 40% misdiagnosis reduction in BraTS and 25% in MVTec) across MNIST, BraTS, and MVTec AD, while remaining compatible with various pre-trained diffusion models. This approach provides a practical, cost-effective strategy to enhance fidelity and reliability of diffusion-based image translation in sensitive domains like medical imaging and automated inspection.

Abstract

Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing "image hallucination" and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a "branching" module generates locally both within and outside OOD regions, and a "fusion" module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models.

Tackling Structural Hallucination in Image Translation with Local Diffusion

TL;DR

This paper addresses structural hallucination in conditional diffusion models when the input contains out-of-distribution (OOD) regions, a problem that risks misdiagnosis in medical imaging and related tasks. It introduces a training-free Local Diffusion framework that first estimates OOD regions, then runs parallel branching denoising for IND and OOD areas and fuses the results to produce coherent outputs, optionally guided by an auxiliary classifier to balance phases. The authors demonstrate significant reductions in hallucinations and improved downstream task performance (e.g., 40% misdiagnosis reduction in BraTS and 25% in MVTec) across MNIST, BraTS, and MVTec AD, while remaining compatible with various pre-trained diffusion models. This approach provides a practical, cost-effective strategy to enhance fidelity and reliability of diffusion-based image translation in sensitive domains like medical imaging and automated inspection.

Abstract

Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing "image hallucination" and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a "branching" module generates locally both within and outside OOD regions, and a "fusion" module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models.
Paper Structure (40 sections, 16 equations, 14 figures, 5 tables)

This paper contains 40 sections, 16 equations, 14 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Models
  4. Out-of-Distribution Generalization
  5. Methodology
  6. Preliminaries: Conditional Denoising Diffusion Probabilistic Models
  7. Hypothesis and Justification
  8. Motivational Experiments: Hypothesis Verification
  9. Can OOD-based Local Image Generation Help to Reduce Hallucination?
  10. The Impact of OOD Segmentation on Hallucinations
  11. Identifying Hallucination Hotspots in Diffusion Models
  12. Proposed Framework
  13. OOD Estimation
  14. Branching Module
  15. Fusion Module
  16. ...and 25 more sections

Figures (14)

  • Figure 1: Traditional conditional diffusion process vs. our OOD/IND Local Diffusion.
  • Figure 2: Examples of structural hallucinations (indicated by arrows) in DDPMddpm compared with the outcomes of our method for BraTS image translation (left) and MVTec super-resolution (right).
  • Figure 3: Impact of different OOD boundaries on predictions.
  • Figure 4: Qualitative and quantitative comparisons of predictions starting from different intermediate time points. We sample noisy GT (Flair) and perform a reverse process from it by conditioning the corresponding T1 image.
  • Figure 5: Schematic diagram of our proposed framework.
  • ...and 9 more figures