Kidney Cancer Detection Using 3D-Based Latent Diffusion Models

TL;DR

This work tackles annotation-efficient detection of kidney abnormalities on contrast-enhanced CT by proposing a 3D latent diffusion pipeline that operates in the latent space of a VQ-GAN and combines DDPM, DDIM, and classifier guidance to generate healthy reconstructions and voxel-wise anomaly maps. Pseudo-labels derived from radiology reports enable weak supervision, and the method processes full 3D volumes rather than slices to capture inter-slice context. While the approach demonstrates feasibility and yields interpretable anomaly maps, its segmentation/detection performance lags behind strong supervised baselines (e.g., nnU-Net, nnDetection), and lesion-size analyses reveal substantial challenges for small and large lesions alike. The study identifies key directions for improvement, including false-positive reduction, improved guidance, and exploring alternative diffusion architectures to push toward annotation-efficient, volumetric anomaly detection in complex abdominal anatomy.

Abstract

In this work, we present a novel latent diffusion-based pipeline for 3D kidney anomaly detection on contrast-enhanced abdominal CT. The method combines Denoising Diffusion Probabilistic Models (DDPMs), Denoising Diffusion Implicit Models (DDIMs), and Vector-Quantized Generative Adversarial Networks (VQ-GANs). Unlike prior slice-wise approaches, our method operates directly on an image volume and leverages weak supervision with only case-level pseudo-labels. We benchmark our approach against state-of-the-art supervised segmentation and detection models. This study demonstrates the feasibility and promise of 3D latent diffusion for weakly supervised anomaly detection. While the current results do not yet match supervised baselines, they reveal key directions for improving reconstruction fidelity and lesion localization. Our findings provide an important step toward annotation-efficient, generative modeling of complex abdominal anatomy.

Figures (2)

• Figure 1.1: Overview of the proposed anomaly detection pipeline. Kidney patches, extracted using TotalSegmentator masks, are encoded into a latent space via a VQ-GAN. The latent patch undergoes $L$ forward (noising) diffusion steps, followed by $L$ reverse (denoising) steps with classifier guidance. The denoised latent patch is decoded to produce a healthy reconstruction. Subtracting this from the original yields the anomaly map.
• Figure 1.2: Visual comparison of diffusion-based reconstructions on four example images.