Interactive Tumor Progression Modeling via Sketch-Based Image Editing

TL;DR

SkEditTumor introduces a sketch-conditioned diffusion framework for controllable tumor progression editing in medical images. It combines a sketch-refinement strategy, a variational autoencoder, and a latent diffusion model conditioned on both sketches and a reference map, with voxel-spacing conditioning to preserve context. The method achieves state-of-the-art fidelity and segmentation accuracy across BraTS, LiTS, KiTS, and MSD-Pancreas datasets, outperforming baselines like T2I-Adapter and DiffTumor. While demonstrated in 2D, the approach provides a versatile tool for visualizing tumor evolution and improving clinical communication, with future work extending to 3D imaging.

Abstract

Accurately visualizing and editing tumor progression in medical imaging is crucial for diagnosis, treatment planning, and clinical communication. To address the challenges of subjectivity and limited precision in existing methods, we propose SkEditTumor, a sketch-based diffusion model for controllable tumor progression editing. By leveraging sketches as structural priors, our method enables precise modifications of tumor regions while maintaining structural integrity and visual realism. We evaluate SkEditTumor on four public datasets - BraTS, LiTS, KiTS, and MSD-Pancreas - covering diverse organs and imaging modalities. Experimental results demonstrate that our method outperforms state-of-the-art baselines, achieving superior image fidelity and segmentation accuracy. Our contributions include a novel integration of sketches with diffusion models for medical image editing, fine-grained control over tumor progression visualization, and extensive validation across multiple datasets, setting a new benchmark in the field.

Figures (4)

• Figure 1: Illustration of the proposed sketch-guided tumor progression modeling approach. The user sketches directly on the tumor region to generate an edited image, simulating potential tumor growth or regression based on the envisioned changes.
• Figure 2: Overflow of the SkEditTumor framework. In Stage 1, the tumor region is segmented, and edge detection is applied to generate edge maps. A deformation module introduces variability to simulate hand-drawn sketch imperfections, followed by a refinement network to produce a precise sketch $\mathbf{S}$. In Stage 2, a VAE-GAN is trained to encode image features with a discriminator ensures realism through adversarial training. In Stage 3, a diffusion model integrates $\mathbf{S}$ and reference map $\mathbf{M}$ into its latent space, progressively denoising it to produce realistic and controllable tumor progression edits.
• Figure 3: Reconstruction and editing results on the BraTS dataset. Qualitative comparison of SkEditTumor (ours), T2I-Adapter, and DiffTumor. Columns 1-3 show reconstruction results with unchanged sketches, including the ground truth, difference maps, and reconstructed images. Columns 4-6 display cropped tumor regions. Columns 7-12 demonstrate tumor progression and regression edits achieved by modifying the sketches, with SkEditTumor delivering more accurate and visually consistent results compared to baselines.
• Figure 4: Tumor editing results across four datasets (BraTS, LiTS, KiTS, MSD-Pancreas). For each dataset, the input image, real tumor segmentation, and zoom-in view of the tumor region are shown, followed by the refined sketch and the corresponding segmentation for tumor expansion. The tumor editing results generated by T2I-Adapter, DiffTumor, and SkEditTumor (ours) are presented in the final three columns. The tumor regions (red boxes) highlight the differences in tumor structure preservation, with SkEditTumor producing more realistic and accurate edits compared to baseline methods.