Single-shot Star-convex Polygon-based Instance Segmentation for Spatially-correlated Biomedical Objects

TL;DR

This work addresses the challenge of nested, spatially-correlated objects in biomedical imaging by introducing HydraStarDist (HSD) and HSD-WBR, two StarDist-based branched architectures that share an encoder to jointly segment inner and outer objects in a single pass. A key contribution is the Within Boundary Regularisation (WBR) penalty, which enforces nesting constraints and reduces false inner predictions outside outer boundaries. The authors propose a task-relevant evaluation metric, Joint TP Rate (JTPR), alongside IoU-based and AP metrics, and demonstrate that HSD and especially HSD-WBR achieve competitive traditional metrics while attaining superior JTPR on HeLaCytoNuc and VACVPlaque datasets, with improved efficiency versus baseline models. The findings suggest spatial-correlation priors can enhance nested object learning and enable efficient single-shot segmentation applicable to diverse biomedical imaging tasks, including partial inclusion/exclusion phenomena in complex interactions. Overall, the work contributes a unified, computationally efficient framework for nested instance segmentation and provides detailed ablations and datasets to support adoption and extension in related domains.

Abstract

Biomedical images often contain objects known to be spatially correlated or nested due to their inherent properties, leading to semantic relations. Examples include cell nuclei being nested within eukaryotic cells and colonies growing exclusively within their culture dishes. While these semantic relations bear key importance, detection tasks are often formulated independently, requiring multi-shot analysis pipelines. Importantly, spatial correlation could constitute a fundamental prior facilitating learning of more meaningful representations for tasks like instance segmentation. This knowledge has, thus far, not been utilised by the biomedical computer vision community. We argue that the instance segmentation of two or more categories of objects can be achieved in parallel. We achieve this via two architectures HydraStarDist (HSD) and the novel (HSD-WBR) based on the widely-used StarDist (SD), to take advantage of the star-convexity of our target objects. HSD and HSD-WBR are constructed to be capable of incorporating their interactions as constraints into account. HSD implicitly incorporates spatial correlation priors based on object interaction through a joint encoder. HSD-WBR further enforces the prior in a regularisation layer with the penalty we proposed named Within Boundary Regularisation Penalty (WBR). Both architectures achieve nested instance segmentation in a single shot. We demonstrate their competitiveness based on $IoU_R$ and AP and superiority in a new, task-relevant criteria, Joint TP rate (JTPR) compared to their baseline SD and Cellpose. Our approach can be further modified to capture partial-inclusion/-exclusion in multi-object interactions in fluorescent or brightfield microscopy or digital imaging. Finally, our strategy suggests gains by making this learning single-shot and computationally efficient.

Figures (8)

• Figure 1: StarDist-based instance segmentation architectures (Section \ref{['sec:methods']}). Architecture diagrams of the instance segmentation model for star-convex objects (a) StarDist, (b) branched architecture based on the SD model HydraStarDist and (c) branched architecture with WBR penalty HydraStarDistWBR (Section \ref{['sec:methods']}).
• Figure 2: Limitations of approach in Delong09. Here object 2 bounds objects 1 and 3 that may either be (a) Exclusive or (b) Partially Overlapping with each other. $\alpha$ is the fraction of overlap between 1 and 3 (Equation \ref{['eq:wbr_loss_limitation_overlap']}).
• Figure 3: HeLaCytoNuc rodare_HeLaCytoNuc24 dataset (Subsection \ref{['subsection:hela_dataset']}). (a) Fluorescence micrograph with cytoplasm and nuclei stains pseudo-coloured red and blue respectively, $x_n$. (b) Annotations of $x_n$ corresponding to the cytoplasm instance mask $y^2_n$ and (c) nuclei instance masks $y^1_n$. Scale bar 200 $\mu$m.
• Figure 4: VACVPlaque rodare_VACVPlaque24 dataset (Subsection \ref{['subsection:plaque_dataset']}). (a) RGB mobile photographs of plaques within 6-well tissue culture plate, $x_n$. (b) Annotations of $x_n$ corresponding to the wells instance mask $y^2_n$ and (c) plaques instance mask $y^1_n$.
• Figure 5: HeLaCytoNuc Single-shot instance segmentation results (Sections \ref{['sec:experiments']} and \ref{['sec:discussion']}), Predicted and Ground Truth (GT) (First row - Cytoplasm, Second row - Nuclei). Results for (a) HSD and (b) HSD-WBR (Section \ref{['sec:methods']}).