Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring

TL;DR

This work analyzes the topological structure of real cell adjacency graphs and finds that they are largely non-bipartite due to dense odd cycles, challenging 2-coloring-based approaches. It introduces Disco, a divide-and-conquer framework that combines Explicit Marking (topology-driven label generation) with Implicit Disambiguation (a constrained loss enforcing feature separation among adjacent instances) to robustly segment densely overlapping cells. The method achieves state-of-the-art performance across four diverse datasets, including the high-density GBC-FS 2025, and provides a Conflict Map as a new interpretability tool for topological complexity. The results support a principled, efficient, and interpretable approach to complex cell instance segmentation with strong generalization capabilities.

Abstract

Accurate cell instance segmentation is foundational for digital pathology analysis. Existing methods based on contour detection and distance mapping still face significant challenges in processing complex and dense cellular regions. Graph coloring-based methods provide a new paradigm for this task, yet the effectiveness of this paradigm in real-world scenarios with dense overlaps and complex topologies has not been verified. Addressing this issue, we release a large-scale dataset GBC-FS 2025, which contains highly complex and dense sub-cellular nuclear arrangements. We conduct the first systematic analysis of the chromatic properties of cell adjacency graphs across four diverse datasets and reveal an important discovery: most real-world cell graphs are non-bipartite, with a high prevalence of odd-length cycles (predominantly triangles). This makes simple 2-coloring theory insufficient for handling complex tissues, while higher-chromaticity models would cause representational redundancy and optimization difficulties. Building on this observation of complex real-world contexts, we propose Disco (Densely-overlapping Cell Instance Segmentation via Adjacency-aware COllaborative Coloring), an adjacency-aware framework based on the "divide and conquer" principle. It uniquely combines a data-driven topological labeling strategy with a constrained deep learning system to resolve complex adjacency conflicts. First, "Explicit Marking" strategy transforms the topological challenge into a learnable classification task by recursively decomposing the cell graph and isolating a "conflict set." Second, "Implicit Disambiguation" mechanism resolves ambiguities in conflict regions by enforcing feature dissimilarity between different instances, enabling the model to learn separable feature representations.

Figures (10)

• Figure : Figure 1. Visual comparison of mainstream instance segmentation paradigms with our proposed Disco framework. (a) Detection-based methods are constrained by coarse bounding box representations and heuristic non-maxima suppression (NMS). (b) Contour-based methods are highly sensitive to binarization thresholds. (c) Distance-based methods rely on complex post-processing for instance reconstruction. (d) Disco framework reformulates the problem by directly modeling the cell adjacency graph, ultimately reconstructing instances through topological decoding.
• Figure : Figure 2. Fundamental topological structures in cell adjacency graphs. (a) Simple, 2-colorable bipartite structures. (b) Non-bipartite structures containing odd-length cycles (e.g., 3-cycles), which induce coloring conflicts. (c) A complex “conflict cluster” formed by interconnected odd cycles, leading to secondary conflicts between adjacent conflict nodes.
• Figure : Figure 3. A Cross-dataset comparative analysis of cell adjacency graph topologies. (a) Node Degree Distributions: Local connectivity varies significantly, from the highly sparse PanNuke to the densely clustered GBC-FS 2025. (b) Odd-Length Cycle Distributions: The prevalence of 3-cycles confirms that most real-world cell graphs are non-bipartite, with complexity peaking in the GBC-FS 2025 dataset. The y-axis is on a logarithmic scale.
• Figure : Figure 4. An overview of the training framework for our proposed Disco method. This framework synergistically integrates (1) data-driven topological analysis, which generates a Disco ground truth map $Y_{Disco}$, encoding both bipartite structures and conflict hotspots; (2) a dual-branch segmentation network, which learns to predict a foundational semantic map $P_{sem}$, and a detailed Disco coloring map $P_{color}$; and (3) a decoupled loss system, which provides targeted supervision. Crucially, the Adjacency Constraint Loss ($\mathcal{L}_{adj}$) leverages the ground truth adjacency graph to enforce feature dissimilarity between all neighboring instances in the continuous probability space, thereby enabling end-to-end constrained optimization.
• Figure : Figure 5. A visual decomposition of the Disco label generation process.