Weakly Supervised Object Segmentation by Background Conditional Divergence

TL;DR

This paper tackles weakly supervised foreground segmentation in data-scarce domains by leveraging counterfactual background blending. A masking network outputs alpha-blending masks to generate counterfactual images, and training optimizes a background-conditioned divergence—specifically an energy-based sliced Wasserstein distance—between counterfactuals and real composites, augmented by a background-only supervision term. Central to the approach is clustering backgrounds into $K$ latent classes and conditioning the divergence on these clusters to avoid degenerate masks. The method, implemented with a U-Net and lightweight encoders, operates without pretrained models and demonstrates strong results on sonar datasets, with reasonable generalization to natural images, and shows promise for broader modality applications. Overall, the approach provides a simple, effective pathway for weakly supervised segmentation in domains where labeled pixel-level data are scarce and pretrained models are inapplicable.

Abstract

As a computer vision task, automatic object segmentation remains challenging in specialized image domains without massive labeled data, such as synthetic aperture sonar images, remote sensing, biomedical imaging, etc. In any domain, obtaining pixel-wise segmentation masks is expensive. In this work, we propose a method for training a masking network to perform binary object segmentation using weak supervision in the form of image-wise presence or absence of an object of interest, which provides less information but may be obtained more quickly from manual or automatic labeling. A key step in our method is that the segmented objects can be placed into background-only images to create realistic images of the objects with counterfactual backgrounds. To create a contrast between the original and counterfactual background images, we propose to first cluster the background-only images and then, during learning, create counterfactual images that blend objects segmented from their original source backgrounds to backgrounds chosen from a targeted cluster. One term in the training loss is the divergence between these counterfactual images and the real object images with backgrounds of the target cluster. The other term is a supervised loss for background-only images. While an adversarial critic could provide the divergence, we use sample-based divergences. We conduct experiments on side-scan and synthetic aperture sonar in which our approach succeeds compared to previous unsupervised segmentation baselines that were only tested on natural images. Furthermore, to show generality we extend our experiments to natural images, obtaining reasonable performance with our method that avoids pretrained networks, generative networks, and adversarial critics. The code for this work can be found at \href{GitHub}{https://github.com/bakerhassan/WSOS}.

Figures (22)

• Figure 1: Right: The counterfactual generation model employs a masking network, $M_\theta$, to process composite images, $(X_{\neg c})$, which contain backgrounds other than $c$. These composite images are combined with corresponding background images, $R_c$, of background $c$. The model generates a mask to separate the object from the background, and alpha-blending (as per equation \ref{['eq:generation_eq']}) is used to combine the composite images and background images. This results in a counterfactual image that retains the foreground object from $X_{\neg c}$ but places it against a different background. In this instance, the counterfactual image preserves the foreground object in $X_{\neg c}$ but is presented against a distinct background. Left: Demonstration of the inputs and the outputs of the proposed framework indicated by the blue box on the right. The first three rows display different sets of background images, where each set belongs to the same background label. Each subsequent pair of rows shows composite images with various backgrounds, excluding the target background (indicated by the two headed arrow). The second row in each pair presents the generated counterfactual images, where the foreground object is pasted onto the target background using the model on the right.
• Figure 2: Segmentation model results on AI4Shipwrecks. Each row corresponds to a different image. From left to right: input images, ground truth masks, estimated masks, and thresholded estimated masks at 0.5. Images sites: Lucinda van Valkenburg.
• Figure 3: Segmentation model results on SAS-Clutter object images. Each column consists of a object snippet image, its estimated segmentation mask, and the masked object.
• Figure 4: Results of the segmentation model for CUB. From top to bottom: input images, estimated masks, true masks, and thresholded estimated masks at 0.5.
• Figure 5: Generated counterfactual images based on test images using 10 clusters. Each block is composed of two rows and 8 columns. Each block shows the original input images in the first row and the corresponding counterfactual images in the second row. The counterfactual are created by alpha-blending the real input images with randomly selected backgrounds. Each second row in the blocks should ideally be composed of the same background clusters. An example of the obtained clusters is shown in Figure \ref{['fig:bg_cub_clusters_10']} in the appendix.