Can We Go Beyond Visual Features? Neural Tissue Relation Modeling for Relational Graph Analysis in Non-Melanoma Skin Histology

TL;DR

This work tackles histopathology tissue segmentation in non-melanoma skin cancer by addressing the limitation of texture-only CNNs that ignore inter-tissue context. It introduces Neural Tissue Relational Modeling (NTRM), a framework that augments CNN features with a Tissue Relation Module (TRM) that builds a tissue-level graph and performs graph-based message passing to refine segmentation. The TRM includes region proposals, per-tissue node embeddings, boundary-aware edge construction, and a multi-layer GNN, with global embeddings for absent tissues, all fused back into the CNN decoder. On a histopathology dataset, NTRM achieves higher mean IoU and Dice scores than baselines, with particularly improved boundary delineation, demonstrating the value of explicit inter-tissue relational modeling for context-aware and interpretable histology segmentation.

Abstract

Histopathology image segmentation is essential for delineating tissue structures in skin cancer diagnostics, but modeling spatial context and inter-tissue relationships remains a challenge, especially in regions with overlapping or morphologically similar tissues. Current convolutional neural network (CNN)-based approaches operate primarily on visual texture, often treating tissues as independent regions and failing to encode biological context. To this end, we introduce Neural Tissue Relation Modeling (NTRM), a novel segmentation framework that augments CNNs with a tissue-level graph neural network to model spatial and functional relationships across tissue types. NTRM constructs a graph over predicted regions, propagates contextual information via message passing, and refines segmentation through spatial projection. Unlike prior methods, NTRM explicitly encodes inter-tissue dependencies, enabling structurally coherent predictions in boundary-dense zones. On the benchmark Histopathology Non-Melanoma Skin Cancer Segmentation Dataset, NTRM outperforms state-of-the-art methods, achieving a robust Dice similarity coefficient that is 4.9\% to 31.25\% higher than the best-performing models among the evaluated approaches. Our experiments indicate that relational modeling offers a principled path toward more context-aware and interpretable histological segmentation, compared to local receptive-field architectures that lack tissue-level structural awareness. Our code is available at https://github.com/shravan-18/NTRM.

Figures (5)

• Figure 1: NTRM framework pipeline showing CNN-based encoding, initial segmentation, TRM module, and final decoding for relationally-informed histological segmentation.
• Figure 2: NTRM architecture. A ResNet18 backbone extracts hierarchical encoder features $\{\mathcal{E}_1,\dots,\mathcal{E}_5\}$, which are decoded into an initial segmentation map. The TRM module receives this map and early decoded features $\mathcal{D}_2$, and refines them via graphical modeling of tissue-type relationships. The final prediction is produced after fusing the refined features with $\mathcal{D}_2$ via deeper decoder layers.
• Figure 3: Pipeline of the TRM module. Initial softmax predictions and early CNN features are used to define tissue-specific regions. Node features are extracted via masked pooling, and edges are created between spatially adjacent regions. A GNN performs message passing over this tissue graph, and refined node embeddings are projected back to the spatial domain. Tissue visuals shown include INF (Inflammation), BKG (Background), RET (Reticular Dermis), and HYP (Hypodermis).
• Figure 4: Qualitative comparison of segmentation results across non-melanoma skin cancer types: BCC, SCC, and IEC. Our method demonstrates improved localization of class boundaries and reduction in false positives (e.g., SCC) compared to others.
• Figure 5: Impact of TRM: Left – initial segmentation from CNN; Center – final prediction after TRM; Right – improvement map overlay (green shows corrected predictions). TRM corrects major errors near BCC-reticular and RET-hypodermis interfaces.