Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Neural Networks for Surgical Scene Segmentation

Yihan Li, Nikhil Churamani, Maria Robu, Imanol Luengo, Danail Stoyanov

TL;DR

This work tackles surgical scene segmentation for hepatocystic anatomy by integrating Vision Transformer features with Graph Neural Networks to explicitly model spatial relationships. It introduces two graph-based models: a static GCNII on a fixed graph and a dynamic GAT with Differentiable Graph Generator for adaptive topology learning. The methods achieve 7–8% improvements in mIoU and around 6% in mDice on Endoscapes-Seg50 and CholecSeg8k, producing anatomically coherent predictions especially for thin, safety-critical structures. By marrying global context with relational reasoning, the approach improves interpretability and reliability, paving the way for safer laparoscopic and robot-assisted surgery; future work includes temporal graphs and real-time deployment.

Abstract

Purpose: Accurate identification of hepatocystic anatomy is critical to preventing surgical complications during laparoscopic cholecystectomy. Deep learning models often struggle with occlusions, long-range dependencies, and capturing the fine-scale geometry of rare structures. This work addresses these challenges by introducing graph-based segmentation approaches that enhance spatial and semantic understanding in surgical scene analyses. Methods: We propose two segmentation models integrating Vision Transformer (ViT) feature encoders with Graph Neural Networks (GNNs) to explicitly model spatial relationships between anatomical regions. (1) A static k Nearest Neighbours (k-NN) graph with a Graph Convolutional Network with Initial Residual and Identity Mapping (GCNII) enables stable long-range information propagation. (2) A dynamic Differentiable Graph Generator (DGG) with a Graph Attention Network (GAT) supports adaptive topology learning. Both models are evaluated on the Endoscapes-Seg50 and CholecSeg8k benchmarks. Results: The proposed approaches achieve up to 7-8% improvement in Mean Intersection over Union (mIoU) and 6% improvement in Mean Dice (mDice) scores over state-of-the-art baselines. It produces anatomically coherent predictions, particularly on thin, rare and safety-critical structures. Conclusion: The proposed graph-based segmentation methods enhance both performance and anatomical consistency in surgical scene segmentation. By combining ViT-based global context with graph-based relational reasoning, the models improve interpretability and reliability, paving the way for safer laparoscopic and robot-assisted surgery through a precise identification of critical anatomical features.

Graph Neural Networks for Surgical Scene Segmentation

TL;DR

This work tackles surgical scene segmentation for hepatocystic anatomy by integrating Vision Transformer features with Graph Neural Networks to explicitly model spatial relationships. It introduces two graph-based models: a static GCNII on a fixed graph and a dynamic GAT with Differentiable Graph Generator for adaptive topology learning. The methods achieve 7–8% improvements in mIoU and around 6% in mDice on Endoscapes-Seg50 and CholecSeg8k, producing anatomically coherent predictions especially for thin, safety-critical structures. By marrying global context with relational reasoning, the approach improves interpretability and reliability, paving the way for safer laparoscopic and robot-assisted surgery; future work includes temporal graphs and real-time deployment.

Abstract

Purpose: Accurate identification of hepatocystic anatomy is critical to preventing surgical complications during laparoscopic cholecystectomy. Deep learning models often struggle with occlusions, long-range dependencies, and capturing the fine-scale geometry of rare structures. This work addresses these challenges by introducing graph-based segmentation approaches that enhance spatial and semantic understanding in surgical scene analyses. Methods: We propose two segmentation models integrating Vision Transformer (ViT) feature encoders with Graph Neural Networks (GNNs) to explicitly model spatial relationships between anatomical regions. (1) A static k Nearest Neighbours (k-NN) graph with a Graph Convolutional Network with Initial Residual and Identity Mapping (GCNII) enables stable long-range information propagation. (2) A dynamic Differentiable Graph Generator (DGG) with a Graph Attention Network (GAT) supports adaptive topology learning. Both models are evaluated on the Endoscapes-Seg50 and CholecSeg8k benchmarks. Results: The proposed approaches achieve up to 7-8% improvement in Mean Intersection over Union (mIoU) and 6% improvement in Mean Dice (mDice) scores over state-of-the-art baselines. It produces anatomically coherent predictions, particularly on thin, rare and safety-critical structures. Conclusion: The proposed graph-based segmentation methods enhance both performance and anatomical consistency in surgical scene segmentation. By combining ViT-based global context with graph-based relational reasoning, the models improve interpretability and reliability, paving the way for safer laparoscopic and robot-assisted surgery through a precise identification of critical anatomical features.

Paper Structure

This paper contains 18 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Image Segmentation
  4. Graph-Based Learning
  5. Node Construction and Feature Extraction
  6. Edge Construction
  7. Graph Neural Networks
  8. Methods
  9. Feature Encoder
  10. Graph Construction
  11. GNN Training
  12. Segmentation Decoder and Loss Function
  13. Experiments
  14. Results and Discussion
  15. Results
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Proposed GCNII-6 and GAT-DGG use ViT-based encoders to construct a graph (for each dataset) offline. GCNII-6 uses a static graph while GAT-DGG dynamically adapts the topology of this graph, both learning to segment different structures.
  • Figure 2: Qualitative Results on Endoscapes-Seg50. Classes: background , cystic plate , calot’s triangle , cystic artery , cystic duct , gallbladder , tool .
  • Figure 3: Qualitative Results on CholecSeg8k. Classes: background , abdominal wall , liver , gastrointestinal tract , fat , grasper , L-hook electrocautery , gallbladder , liver ligament .
  • Figure 4: Visualisation on Endoscapes-Seg50 of a cystic duct node learning long-range dependencies across neighbourhood nodes. Classes: background , cystic plate , calot’s triangle , cystic artery , cystic duct , gallbladder , tool .