BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation

TL;DR

BackSplit tackles the persistent challenge of small-lesion segmentation by treating the background as a spectrum of semantically meaningful auxiliary classes rather than a single background. The authors provide an information-theoretic foundation showing that multiclass supervision increases the expected Fisher Information for the target lesion, yielding more efficient and stable MLEs than traditional binary training. Empirically, BackSplit delivers consistent gains across five diverse datasets, three architectures, and a variety of auxiliary-label sources, including automatically generated organ masks and interactive/semi-automatic cues, with minimal parameter overhead. The approach is architecture-agnostic, scalable, and robust to label noise, suggesting broad applicability in clinical settings for improved lesion delineation and reduced false positives. The work also outlines practical guidance for selecting auxiliary structures and demonstrates performance improvements under fine-tuning and partial supervision scenarios.

Abstract

Segmenting small lesions in medical images remains notoriously difficult. Most prior work tackles this challenge by either designing better architectures, loss functions, or data augmentation schemes; and collecting more labeled data. We take a different view, arguing that part of the problem lies in how the background is modeled. Common lesion segmentation collapses all non-lesion pixels into a single "background" class, ignoring the rich anatomical context in which lesions appear. In reality, the background is highly heterogeneous-composed of tissues, organs, and other structures that can now be labeled manually or inferred automatically using existing segmentation models. In this paper, we argue that training with fine-grained labels that sub-divide the background class, which we call BackSplit, is a simple yet powerful paradigm that can offer a significant performance boost without increasing inference costs. From an information theoretic standpoint, we prove that BackSplit increases the expected Fisher Information relative to conventional binary training, leading to tighter asymptotic bounds and more stable optimization. With extensive experiments across multiple datasets and architectures, we empirically show that BackSplit consistently boosts small-lesion segmentation performance, even when auxiliary labels are generated automatically using pretrained segmentation models. Additionally, we demonstrate that auxiliary labels derived from interactive segmentation frameworks exhibit the same beneficial effect, demonstrating its robustness, simplicity, and broad applicability.

Figures (8)

• Figure 1: Comparison between conventional binary segmentation (Model A) and our BackSplit paradigm (Model B). Conventional lesion segmentation collapses all non-lesion regions into a single background, discarding the anatomical context and often producing false positives (yielding a Dice overlap score of 0.0). In contrast, BackSplit refines the background into semantically meaningful auxiliary structures (e.g., organ parenchyma) that are learned jointly with the lesion target. This structured background supervision enriches contextual understanding, yielding sharper lesion boundaries, fewer false detections (e.g., Dice = 0.72), and theoretically more stable predictions—consistent with higher expected Fisher Information and reduced estimator variance compared to conventional binary training.
• Figure 2: (Left) In binary training, the background gradient $s(\text{Background})$ conflates signals from multiple anatomical structures. Decomposing it into semantically distinct support structures (e.g., kidney) yields more disentangled and informative gradients. (Right) The corresponding log-likelihood level sets show that the decomposed formulation ($\mathcal{I}_Y$, blue) has sharper curvature and higher Fisher Information than the collapsed binary case ($\mathcal{I}_Z$, pink), leading to tighter and more stable parameter estimates.
• Figure 3: (Left) Fine-tuning a pretrained binary model with auxiliary structures steadily improves performance across epochs. (Right) Under partial support supervision, limited auxiliary data initially reduce performance but later yield consistent gains as more auxiliary structures are added, approaching full BackSplit performance. Evaluated on KiTS23 (Target is Cyst).
• Figure 4: Adding even a single auxiliary structure yields an immediate improvement over regular training, and although incorporating multiple structures introduces mixed behavior with both gains and small drops, performance remains consistently well above the regular-training baseline. This experiment is conducted on the AutoPET dataset by incrementally adding auxiliary classes from AbdomenAtlas1.0Mini using a U-Net backbone.
• Figure 5: (Top) Effect of fine-tuning with BackSplit on a pretrained binary model for kidney cyst segmentation (KiTS23). Mean Dice steadily improves with training epochs, approaching full BackSplit performance. (Bottom) Similar trend observed for pancreatic tumor segmentation (PANTHER-MR), where fine-tuning progressively narrows the gap between regular and full BackSplit.

Theorems & Definitions (8)

• Lemma 1: Score Projection louis1982findingoakes1999direct
• Theorem 1: Label coarsening reduces expected Fisher information.
• Corollary 1: Asymptotic Efficiency of the Multiclass MLE.
• Proposition 1: Softmax Expected Fisher Information Decomposition.
• Lemma 1: Score Projection louis1982findingoakes1999direct
• Theorem 1: Label coarsening reduces expected Fisher information.
• Corollary 1: Asymptotic Efficiency of the Multiclass MLE.
• Proposition 1: Softmax Expected Fisher Information Decomposition.