An inpainting approach to manipulate asymmetry in pre-operative breast images

TL;DR

This work tackles predicting the aesthetic outcomes of breast cancer treatment by simulating post-operative asymmetries on pre-operative breast images using inpainting. It introduces both non-invertible symmetry-aware inpainting (Double GAN with two encoders) and invertible inpainting (Glow/i-RevNet) to inpaint and segment breasts, enabling manipulation even when annotations are unavailable. Transferring asymmetries relies on ratio-based constraints to reposition the nipple and lower contour, formalized by relations such as $y_{tar} = y_{oth} \\frac{y'_{tar}}{y'_{oth}}$ and $\\mathcal{D}(x) = \\frac{x_{ext} - x_{nip}}{x_{ext} - x_{int}}$, with $\\frac{\\mathcal{D}(x_{tar})}{\\mathcal{D}(x_{oth})} = \\frac{\\mathcal{D}(x'_{tar})}{\\mathcal{D}(x'_{oth})}$. evaluations on CP and BC datasets show the proposed methods achieve realistic manipulations and competitive transfer of asymmetries, surpassing prior approaches in image realism while highlighting artifacts at stronger deformations. The results support a practical pathway for patient-specific outcome visualization and treatment planning, with future work aimed at semi-supervised training and broader post-surgical attributes such as scars and skin color changes.

Abstract

One of the most frequent modalities of breast cancer treatment is surgery. Breast surgery can cause visual alterations to the breasts, due to scars and asymmetries. To enable an informed choice of treatment, the patient must be adequately informed of the aesthetic outcomes of each treatment plan. In this work, we propose an inpainting approach to manipulate breast shape and nipple position in breast images, for the purpose of predicting the aesthetic outcomes of breast cancer treatment. We perform experiments with various model architectures for the inpainting task, including invertible networks capable of manipulating breasts in the absence of ground-truth breast contour and nipple annotations. Experiments on two breast datasets show the proposed models' ability to realistically alter a patient's breasts, enabling a faithful reproduction of breast asymmetries of post-operative patients in pre-operative images.

Figures (11)

• Figure 1: Generic pipeline to transfer breast asymmetries from post-operative patients into pre-operative images.
• Figure 2: Generic overview of inpainting framework.
• Figure 3: Proposed Double GAN model. During training the model receives the masked right breast and a flipped version of the left breast, and is trained to predict the right breast. On inference, it receives the masked and original versions of the right breast to inpaint the breast while preserving its color progression.
• Figure 4: Overview of invertible inpainting approach for manipulating breast asymmetries without requiring annotations. During training, the segmentation direction receives the original image and predicts its masked version, and the inpainting direction receives the masked image and predicts the original. During inference, the original image goes through the segmentation direction to obtain its masked version. Then, the post-processing pipeline improves the quality of the masked image and repositions the nipple and breast. Finally, the inpainting direction obtains the morphed image.
• Figure 5: Pipeline to transfer breast asymmetries from post-operative patients into pre-operative images using inpainting networks.