Region-Normalized DPO for Medical Image Segmentation under Noisy Judges

TL;DR

Medical image segmentation often hinges on costly pixel-wise annotations, motivating learning from inexpensive, noisy QC-like judge signals. The authors introduce Region-Normalized DPO (RN-DPO), which normalizes preference updates to the region of disagreement between candidate masks, reducing the impact of misleading comparisons. Through experiments on JSRT and ACDC across multiple base/judge regimes, RN-DPO yields higher peak IoU and better training stability than vanilla DPO and strong baselines, including under an oracle judge. The approach enables more robust, scalable segmentation refinement without additional pixel annotations, highlighting a general design principle for learning from imperfect comparative feedback in medical imaging.

Abstract

While dense pixel-wise annotations remain the gold standard for medical image segmentation, they are costly to obtain and limit scalability. In contrast, many deployed systems already produce inexpensive automatic quality-control (QC) signals like model agreement, uncertainty measures, or learned mask-quality scores which can be used for further model training without additional ground-truth annotation. However, these signals can be noisy and biased, making preference-based fine-tuning susceptible to harmful updates. We study Direct Preference Optimization (DPO) for segmentation from such noisy judges using proposals generated by a supervised base segmenter trained on a small labeled set. We find that outcomes depend strongly on how preference pairs are mined: selecting the judge's top-ranked proposal can improve peak performance when the judge is reliable, but can amplify harmful errors under weaker judges. We propose Region-Normalized DPO (RN-DPO), a segmentation-aware objective which normalizes preference updates by the size of the disagreement region between masks, reducing the leverage of harmful comparisons and improving optimization stability. Across two medical datasets and multiple regimes, RN-DPO improves sustained performance and stabilizes preference-based fine-tuning, outperforming standard DPO and strong baselines without requiring additional pixel annotations.

Figures (5)

• Figure 1: Main pipeline. We fine-tune a base segmenter $\pi_{\theta}$ on unlabeled images using comparative feedback. For each image, $K$ candidate masks are generated, scored by a judge, and converted into preference pairs by a miner. The base segmenter is then updated with RN-DPO, which normalizes the preference update over the disagreement region between the two masks instead of the whole image. Notation follows Sec. \ref{['sec:method']}.
• Figure 2: Validation IoU curves. JSRT (top) and ACDC (bottom) shown with weak/strong judge regimes for the weak base model.
• Figure 3: Harmful update mass over training (JSRT, weak base/weak judge). We track the normalized harmful update mass $H$ (Eq. \ref{['eq:harm_mass']}), which aggregates oracle-negative preference margins weighted by disagreement fraction and DPO update strength.
• Figure 4: Qualitative segmentation results on JSRT. Examples comparing the output segmentations from the base segmenter with models fine-tuned using DPO and RN-DPO for the weak base regime.
• Figure 5: Qualitative segmentation proposals using different strategies on JSRT. (SDF = signed distance fields)