RadiomicsFill-Mammo: Synthetic Mammogram Mass Manipulation with Radiomics Features

TL;DR

This work tackles data scarcity and labeling costs in mammography by proposing RadiomicsFill-Mammo, a stable-diffusion inpainting framework that generates synthetic mammogram masses conditioned on radiomics features and clinical variables. A tabular encoder based on the MET approach learns feature-specific embeddings to guide mass generation, while leveraging the opposite breast input to improve inpainting realism. The method achieves radiomics feature-consistent generations and high-quality synthetic masses, and demonstrably boosts downstream mass-detection performance when used as data augmentation; it also shows potential for adaptation to external datasets via LoRa-based fine-tuning. Overall, RadiomicsFill-Mammo provides a clinically meaningful, data-efficient approach to simulate tumors with controllable attributes, with implications for treatment planning and broader medical-imaging research across modalities.

Abstract

Motivated by the question, "Can we generate tumors with desired attributes?'' this study leverages radiomics features to explore the feasibility of generating synthetic tumor images. Characterized by its low-dimensional yet biologically meaningful markers, radiomics bridges the gap between complex medical imaging data and actionable clinical insights. We present RadiomicsFill-Mammo, the first of the RadiomicsFill series, an innovative technique that generates realistic mammogram mass images mirroring specific radiomics attributes using masked images and opposite breast images, leveraging a recent stable diffusion model. This approach also allows for the incorporation of essential clinical variables, such as BI-RADS and breast density, alongside radiomics features as conditions for mass generation. Results indicate that RadiomicsFill-Mammo effectively generates diverse and realistic tumor images based on various radiomics conditions. Results also demonstrate a significant improvement in mass detection capabilities, leveraging RadiomicsFill-Mammo as a strategy to generate simulated samples. Furthermore, RadiomicsFill-Mammo not only advances medical imaging research but also opens new avenues for enhancing treatment planning and tumor simulation. Our code is available at https://github.com/nainye/RadiomicsFill.

Figures (8)

• Figure 1: Overview of the RadiomicsFill-Mammo model, a stable diffusion-based architecture. This model performs iterative denoising for tumor inpainting within masked regions on a noisy latent vector, utilizing information from unmasked regions, the opposite breast, and specific tumor conditions.
• Figure 2: Pretraining Process of the Tabular Encoder. Unmasked feature values (i.e., radiomics features) and their embeddings are concatenated for the encoder input. For masked features, a mask token and the masked feature's embeddings are concatenated, forming the input for the decoder, which focuses on reconstructing tumor conditions. Following pretraining, the encoder is frozen, and its output is utilized for tumor inpainting.
• Figure 3: Different Configurations of Prompt Encoders. Each configuration employs a prompt encoder specialized for its prompt type. Configurations (a) MassTextFill, (b) ClinicaltextFill, and (d) RadiomicsFill-MET utilize pretrained and frozen prompt encoders, whereas the encoder for (c) RadiomicsFill-MLP consists of three MLP layers and is trained alongside the generator.
• Figure 4: Radiomics Feature Consistency Between Real and Synthetic Masses. (a) Grouped violin plots for shape, histogram, and texture (GLCM and GLSZM) features, contrasting real (left) and synthetic (right) distributions. (b) Visual comparison of real and synthetic masses, with red arrows highlighting masses and histograms visualizing the 67 extracted radiomics features underneath.
• Figure 5: Progressive Shape Variation in Synthetic Tumors. Starting with an image containing a mass, we use a normal prompt to replace the mass area with normal tissue. Then, at random positions, we generate synthetic tumors of increasing size, keeping histogram and texture features consistent. Red arrows in each image point to the mass. Below each image, histograms display the radiomics features extracted from the tumor areas.