Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis

TL;DR

The paper addresses the challenge of generating realistic counterfactual medical images with structure-specific interventions. It introduces Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT), which uses pre-trained segmentation models during training to estimate structure-area variables from counterfactual images, guiding the deep structural causal model (DSCM) to produce locally coherent edits on scalar variables. Compared to regressor-based counterfactual fine-tuning (Reg-CFT), Seg-CFT yields more targeted, anatomy-consistent changes and fewer unintended global effects, demonstrated on chest radiographs (PadChest) and coronary artery disease progression (CCTA). The approach preserves simple scalar interventions while improving anatomical consistency, with practical implications for data augmentation, debiasing, disease modeling, and explainability in medical imaging.

Abstract

Counterfactual image generation is a powerful tool for augmenting training data, de-biasing datasets, and modeling disease. Current approaches rely on external classifiers or regressors to increase the effectiveness of subject-level interventions (e.g., changing the patient's age). For structure-specific interventions (e.g., changing the area of the left lung in a chest radiograph), we show that this is insufficient, and can result in undesirable global effects across the image domain. Previous work used pixel-level label maps as guidance, requiring a user to provide hypothetical segmentations which are tedious and difficult to obtain. We propose Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT), which preserves the simplicity of intervening on scalar-valued, structure-specific variables while producing locally coherent and effective counterfactuals. We demonstrate the capability of generating realistic chest radiographs, and we show promising results for modeling coronary artery disease. Code: https://github.com/biomedia-mira/seg-cft.

Figures (3)

• Figure 1: A schematic of the proposed Seg-CFT method, where we utilise pre-trained segmentors to guide counterfactual fine-tuning of DSCMs.
• Figure 2: Generated counterfactuals (CFs) with (a) Reg-CFT and (b) Seg-CFT. First rows show original image $\mathbf{x}$ and CFs $\mathbf{\widetilde{x}}$ with segmentations for left lung (red), right lung (green) and heart (blue). The intervened structure is highlighted with thicker lines. Second rows show direct effect of CFs, i.e. $\mathbf{\widetilde{x}}-\mathbf{x}$. We also report the predicted areas ($px^2$) by segmentors on the bottom. We observe that Seg-CFT produces more locally coherent and spatially consistent interventions.
• Figure 3: Generated CFs with (a) Reg-CFT and (b) Seg-CFT. Left columns show original image $\mathbf{x}$ and CFs $\mathbf{\widetilde{x}}$; right columns show direct effect of CFs, i.e. $\mathbf{\widetilde{x}}-\mathbf{x}$. Seg-CFT produces more locally coherent and spatially consistent interventions. Green arrows indicate expected local changes in plaque with Seg-CFT, while red arrows highlight undesirable changes of non-target structures with Reg-CFT.