A Bayesian Approach to Segmentation with Noisy Labels via Spatially Correlated Distributions

TL;DR

This work tackles semantic segmentation under spatially correlated label noise by introducing the ECCD framework, which models the discrete label errors through a continuous Gaussian latent field $m{ abla}$ with a KMS covariance. The approach derives an ELBO-based objective that yields a cross-entropy term with soft labels and KL regularization, enabling tractable variational inference for high-dimensional, spatially correlated discrete variables. By leveraging the KMS covariance, the method achieves efficient ELBO computation, with the inverse being sparse and the determinant analytic, allowing $O(HW)$ scaling in practice. Empirical results on JSRT (medical) and WHU Building (remote sensing) demonstrate enhanced robustness to moderate-to-high spatial noise, with performance close to clean-label training in some cases and clear advantages over baselines that ignore spatial correlations. The ECCD framework offers a principled, scalable path for handling correlated label errors in segmentation and other structured discrete-variable problems, with potential extensions to 3D and temporal domains.

Abstract

In semantic segmentation, the accuracy of models heavily depends on the high-quality annotations. However, in many practical scenarios, such as medical imaging and remote sensing, obtaining true annotations is not straightforward and usually requires significant human labor. Relying on human labor often introduces annotation errors, including mislabeling, omissions, and inconsistency between annotators. In the case of remote sensing, differences in procurement time can lead to misaligned ground-truth annotations. These label errors are not independently distributed, and instead usually appear in spatially connected regions where adjacent pixels are more likely to share the same errors. To address these issues, we propose an approximate Bayesian estimation based on a probabilistic model that assumes training data include label errors, incorporating the tendency for these errors to occur with spatial correlations between adjacent pixels. However, Bayesian inference for such spatially correlated discrete variables is notoriously intractable. To overcome this fundamental challenge, we introduce a novel class of probabilistic models, which we term the ELBO-Computable Correlated Discrete Distribution (ECCD). By representing the discrete dependencies through a continuous latent Gaussian field with a Kac-Murdock-Szegö (KMS) structured covariance, our framework enables scalable and efficient variational inference for problems previously considered computationally prohibitive. Through experiments on multiple segmentation tasks, we confirm that leveraging the spatial correlation of label errors significantly improves performance. Notably, in specific tasks such as lung segmentation, the proposed method achieves performance comparable to training with clean labels under moderate noise levels. Code is available at https://github.com/pfnet-research/Bayesian_SpatialCorr.

Figures (4)

• Figure 1: (a) Graphical model of our proposed method, the ECCD. The model introduces a latent Gaussian variable $\bm{\eta}$ that encodes spatial correlations of label error among adjacent pixels, thereby enabling a more realistic modeling of the dependency between pixels. (b) Conceptual illustration of spatial correlation. Examples 1 and 2 show how varying $\rho$ affects the correlation strength between the center pixel and its neighbors (top, bottom, sides, and diagonals). When adjacent pixels are likely to contain label errors and spatial correlation is strong ($\rho$ is high) label errors can be inferred with high confidence, even if they are rare.
• Figure 2: Examples of Noisy Labels. In medical imaging, label noise is often simulated using morphological operations such as erosion, dilation, and affine transformations li2021superpixel. Conversely, building segmentation commonly involves omission noise, commission errors, and boundary noise song2022self. The pink region indicates labels present in the noisy but absent in the clean label, while the blue region indicates the opposite.
• Figure 3: Segmentation results on JSRT and WHU Building dataset. The first left three columns show the segmentation results for three classes—Lung, Heart, and Clavicle—in the JSRT dataset, while the right column presents the binary segmentation results for WHU Building dataset in a bar chart. The rows correspond to three different noise intensity settings tested on each dataset, with Dice scores reported for JSRT and IoU scores for WHU Building dataset.
• Figure 4: (a) Effect of $\rho$. Segmentation performance was evaluated across datasets based on $\rho$, which represents the degree of spatial correlation, affects segmentation performance across datasets. The results show that incorporating moderate spatial correlation enhances segmentation performance. Dice scores are reported for the Lung, Heart, and Clavicle, and IoU is used for the WHU Building dataset. (b) Estimated label error. This figure visualizes the label error probability, computed as the sigmoid of $\bm{m}$, the posterior mean of the logit of label error $\bm{\eta}$. As shown in (a), higher accuracy in estimating $\bm{\eta}$ leads to better segmentation performance.