Uncertainty-aware retinal layer segmentation in OCT through probabilistic signed distance functions

TL;DR

The paper tackles the challenge of segmenting ultra-thin retinal layers in OCT scans under noise and pathology by proposing a probabilistic signed distance function (pSDF) that represents boundaries as the zero level set of an SDF and models uncertainty with a Gaussian likelihood. By enforcing the Eikonal constraint on the SDF and exporting the boundary via a soft-boundary extraction, the method yields robust, geometrically grounded segmentations. The authors demonstrate substantial MAE improvements over pixel-wise and regression baselines on internal AMD/controls data and an external Rotterdam dataset, along with meaningful, pathology-aware uncertainty estimates that outperform standard uncertainty methods like MCDO and Deep Ensemble. This work advances reliable, uncertainty-aware retinal layer segmentation and offers a practical biomarker-capable framework for disease progression analysis, with code available at the published GitHub repository.

Abstract

In this paper, we present a new approach for uncertainty-aware retinal layer segmentation in Optical Coherence Tomography (OCT) scans using probabilistic signed distance functions (SDF). Traditional pixel-wise and regression-based methods primarily encounter difficulties in precise segmentation and lack of geometrical grounding respectively. To address these shortcomings, our methodology refines the segmentation by predicting a signed distance function (SDF) that effectively parameterizes the retinal layer shape via level set. We further enhance the framework by integrating probabilistic modeling, applying Gaussian distributions to encapsulate the uncertainty in the shape parameterization. This ensures a robust representation of the retinal layer morphology even in the presence of ambiguous input, imaging noise, and unreliable segmentations. Both quantitative and qualitative evaluations demonstrate superior performance when compared to other methods. Additionally, we conducted experiments on artificially distorted datasets with various noise types-shadowing, blinking, speckle, and motion-common in OCT scans to showcase the effectiveness of our uncertainty estimation. Our findings demonstrate the possibility to obtain reliable segmentation of retinal layers, as well as an initial step towards the characterization of layer integrity, a key biomarker for disease progression. Our code is available at \url{https://github.com/niazoys/RLS_PSDF}.

Figures (11)

• Figure 1: Left: The zero level set (green) of a signed distance function (SDF) $d$ parametrizes a retinal layer in a OCT B-scan. Right: Due to the Eikonal constraint $\lVert \tfrac{\partial}{\partial y} d(x,y) \rVert = 1$, uncertainty in the SDF $d$, here represented as displacement $\Delta d = \sigma$ in the level, induces an equal displacement $\Delta y$ of the contour.
• Figure 2: Layer segmentation performance summary on internal and external test set.
• Figure 3: Segmentation examples for REGR, pREGR, SDF, and pSDF. The first and second row contains structural deformation due to AMD, and the third row presents a case of closely packed layers. Finally, last two rows show instances of Pigment Epithelial Detachment
• Figure 4: (a) Avg. per A-scan variance with pSDF for different types of noise and their non-noisy counterparts, (b) Uncertainty on pathological and noisy conditions with pSDF. The dotted line shows a +1 or -1 standard deviation.
• Figure 5: Schamatics for regression-based and our approach.